[Skip to Navigation]
Sign In
Figure 1.  Flow Figure of Study Eligibility Assessment and Inclusion
Flow Figure of Study Eligibility Assessment and Inclusion
Figure 2.  Publication of Antibiotic and Probiotic Studies Based on Funding Source, Blinding, and Multicenter Design
Publication of Antibiotic and Probiotic Studies Based on Funding Source, Blinding, and Multicenter Design
Table 1.  Characteristics of Eligible Trials
Characteristics of Eligible Trials
Table 2.  Adjusted ORs for the Association Between Publication Status and Trial
Adjusted ORs for the Association Between Publication Status and Trial
1.
Caselli  M, Cassol  F, Calò  G, Holton  J, Zuliani  G, Gasbarrini  A.  Actual concept of “probiotics”: is it more functional to science or business?   World J Gastroenterol. 2013;19(10):1527-1540. doi:10.3748/wjg.v19.i10.1527 PubMedGoogle ScholarCrossref
2.
Freedman  SB, Schnadower  D, Tarr  PI.  The probiotic conundrum: regulatory confusion, conflicting studies, and safety concerns.   JAMA. 2020;323(9):823-824. doi:10.1001/jama.2019.22268 PubMedGoogle ScholarCrossref
3.
Foligné  B, Daniel  C, Pot  B.  Probiotics from research to market: the possibilities, risks and challenges.   Curr Opin Microbiol. 2013;16(3):284-292. doi:10.1016/j.mib.2013.06.008 PubMedGoogle ScholarCrossref
4.
Day  RL, Harper  AJ, Woods  RM, Davies  OG, Heaney  LM.  Probiotics: current landscape and future horizons.   Future Sci OA. 2019;5(4):FSO391. doi:10.4155/fsoa-2019-0004 PubMedGoogle Scholar
5.
Martinelli  M, Banderali  G, Bobbio  M,  et al.  Probiotics’ efficacy in paediatric diseases: which is the evidence? a critical review on behalf of the Italian Society of Pediatrics.   Ital J Pediatr. 2020;46(1):104. doi:10.1186/s13052-020-00862-z PubMedGoogle ScholarCrossref
6.
Khalesi  S, Bellissimo  N, Vandelanotte  C, Williams  S, Stanley  D, Irwin  C.  A review of probiotic supplementation in healthy adults: helpful or hype?   Eur J Clin Nutr. 2019;73(1):24-37. doi:10.1038/s41430-018-0135-9 PubMedGoogle ScholarCrossref
7.
Islam  SU.  Clinical uses of probiotics.   Medicine (Baltimore). 2016;95(5):e2658. doi:10.1097/MD.0000000000002658 PubMedGoogle Scholar
8.
Ciorba  MA.  A gastroenterologist’s guide to probiotics.   Clin Gastroenterol Hepatol. 2012;10(9):960-968. doi:10.1016/j.cgh.2012.03.024 PubMedGoogle ScholarCrossref
9.
Freedman  SB, Williamson-Urquhart  S, Farion  KJ,  et al; PERC PROGUT Trial Group.  Multicenter trial of a combination probiotic for children with gastroenteritis.   N Engl J Med. 2018;379(21):2015-2026. doi:10.1056/NEJMoa1802597 PubMedGoogle ScholarCrossref
10.
Baldassarre  ME, Di Mauro  A, Tafuri  S,  et al.  Effectiveness and safety of a probiotic-mixture for the treatment of infantile colic: a double-blind, randomized, placebo-controlled clinical trial with fecal real-time PCR and NMR-based metabolomics analysis.   Nutrients. 2018;10(2):E195. doi:10.3390/nu10020195 PubMedGoogle Scholar
11.
Marchand  V.  Using probiotics in the paediatric population.   Paediatr Child Health. 2012;17(10):575-576. doi:10.1093/pch/17.10.575 PubMedGoogle ScholarCrossref
12.
Thomas  DW, Greer  FR; American Academy of Pediatrics Committee on Nutrition; American Academy of Pediatrics Section on Gastroenterology, Hepatology, and Nutrition.  Probiotics and prebiotics in pediatrics.   Pediatrics. 2010;126(6):1217-1231. doi:10.1542/peds.2010-2548 PubMedGoogle ScholarCrossref
13.
Huang  R, Hu  J.  Positive effect of probiotics on constipation in children: a systematic review and meta-analysis of six randomized controlled trials.   Front Cell Infect Microbiol. 2017;7:153. doi:10.3389/fcimb.2017.00153 PubMedGoogle ScholarCrossref
14.
Goldenberg  JZ, Yap  C, Lytvyn  L,  et al.  Probiotics for the prevention of Clostridium difficile–associated diarrhea in adults and children.   Cochrane Database Syst Rev. 2017;12:CD006095. doi:10.1002/14651858.CD006095.pub4 PubMedGoogle Scholar
15.
Wojtyniak  K, Szajewska  H.  Systematic review: probiotics for functional constipation in children.   Eur J Pediatr. 2017;176(9):1155-1162. doi:10.1007/s00431-017-2972-2 PubMedGoogle ScholarCrossref
16.
Besselink  MG, van Santvoort  HC, Buskens  E,  et al; Dutch Acute Pancreatitis Study Group.  Probiotic prophylaxis in predicted severe acute pancreatitis: a randomised, double-blind, placebo-controlled trial.   Lancet. 2008;371(9613):651-659. doi:10.1016/S0140-6736(08)60207-X PubMedGoogle ScholarCrossref
17.
van den Nieuwboer  M, van de Burgwal  LHM, Claassen  E.  A quantitative key-opinion-leader analysis of innovation barriers in probiotic research and development: valorisation and improving the tech transfer cycle.   PharmaNutrition. 2016;4(1):9-18. doi:10.1016/j.phanu.2015.09.003Google ScholarCrossref
18.
Allen  SJ, Martinez  EG, Gregorio  GV, Dans  LF.  Probiotics for treating acute infectious diarrhoea.   Cochrane Database Syst Rev. 2010(11):CD003048. doi:10.1002/14651858.CD003048.pub3PubMedGoogle Scholar
19.
Collinson  S, Deans  A, Padua-Zamora  A,  et al.  Probiotics for treating acute infectious diarrhoea.   Cochrane Database Syst Rev. 2020;12(12):CD003048. doi:10.1002/14651858.CD003048.pub4PubMedGoogle Scholar
20.
Sterne  JA, Sutton  AJ, Ioannidis  JP,  et al.  Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials.   BMJ. 2011;343:d4002. doi:10.1136/bmj.d4002 PubMedGoogle ScholarCrossref
21.
Bellomo  R, Warrillow  SJ, Reade  MC.  Why we should be wary of single-center trials.   Crit Care Med. 2009;37(12):3114-3119. doi:10.1097/CCM.0b013e3181bc7bd5 PubMedGoogle ScholarCrossref
22.
Ekmekci  PE.  An increasing problem in publication ethics: publication bias and editors’ role in avoiding it.   Med Health Care Philos. 2017;20(2):171-178. doi:10.1007/s11019-017-9767-0 PubMedGoogle ScholarCrossref
23.
Ross  JS, Tse  T, Zarin  DA, Xu  H, Zhou  L, Krumholz  HM.  Publication of NIH funded trials registered in ClinicalTrials.gov: cross sectional analysis.   BMJ. 2012;344:d7292. doi:10.1136/bmj.d7292 PubMedGoogle ScholarCrossref
24.
Lundh  A, Lexchin  J, Mintzes  B, Schroll  JB, Bero  L.  Industry sponsorship and research outcome.   Cochrane Database Syst Rev. 2017;2:MR000033. doi:10.1002/14651858.MR000033.pub3PubMedGoogle Scholar
25.
Roddick  AJ, Chan  FTS, Stefaniak  JD, Zheng  SL.  Discontinuation and non-publication of clinical trials in cardiovascular medicine.   Int J Cardiol. 2017;244:309-315. doi:10.1016/j.ijcard.2017.06.020 PubMedGoogle ScholarCrossref
26.
DeAngelis  CD, Drazen  JM, Frizelle  FA,  et al; International Committee of Medical Journal Editors.  Clinical trial registration: a statement from the International Committee of Medical Journal Editors.   JAMA. 2004;292(11):1363-1364. doi:10.1001/jama.292.11.1363 PubMedGoogle ScholarCrossref
27.
Huser  V, Cimino  JJ.  Evaluating adherence to the International Committee of Medical Journal Editors’ policy of mandatory, timely clinical trial registration.   J Am Med Inform Assoc. 2013;20(e1):e169-e174. doi:10.1136/amiajnl-2012-001501 PubMedGoogle ScholarCrossref
28.
Hersh  AL, Shapiro  DJ, Pavia  AT, Shah  SS.  Antibiotic prescribing in ambulatory pediatrics in the United States.   Pediatrics. 2011;128(6):1053-1061. doi:10.1542/peds.2011-1337 PubMedGoogle ScholarCrossref
29.
Youngster  I, Avorn  J, Belleudi  V,  et al.  Antibiotic use in children: a cross-national analysis of 6 countries.   J Pediatr. 2017;182:239-244.e1. doi:10.1016/j.jpeds.2016.11.027 PubMedGoogle ScholarCrossref
30.
Bourgeois  FT, Murthy  S, Mandl  KD.  Outcome reporting among drug trials registered in ClinicalTrials.gov.   Ann Intern Med. 2010;153(3):158-166. doi:10.7326/0003-4819-153-3-201008030-00006 PubMedGoogle ScholarCrossref
31.
HeI H. Journal Citation Reports 2019 JCR, based on 2018 data (Clarivate Analytics, Thomson Reuters). 2020. Accessed September 7, 2020. https://mjl.clarivate.com/search-results
32.
Saha  S, Saint  S, Christakis  DA.  Impact factor: a valid measure of journal quality?   J Med Libr Assoc. 2003;91(1):42-46.PubMedGoogle Scholar
33.
Lenth  R. Java applets for power and sample size. 2018. Accessed October 8, 2020. https://stat.uiowa.edu/~rlenth/Power
34.
Landis  JR, Koch  GG.  The measurement of observer agreement for categorical data.   Biometrics. 1977;33(1):159-174. doi:10.2307/2529310 PubMedGoogle ScholarCrossref
35.
Videlock  EJ, Cremonini  F.  Meta-analysis: probiotics in antibiotic-associated diarrhoea.   Aliment Pharmacol Ther. 2012;35(12):1355-1369. doi:10.1111/j.1365-2036.2012.05104.x PubMedGoogle ScholarCrossref
36.
Ritchie  ML, Romanuk  TN.  A meta-analysis of probiotic efficacy for gastrointestinal diseases.   PLoS One. 2012;7(4):e34938. doi:10.1371/journal.pone.0034938 PubMedGoogle Scholar
37.
Amiri  AR, Kanesalingam  K, Cro  S, Casey  AT.  Does source of funding and conflict of interest influence the outcome and quality of spinal research?   Spine J. 2014;14(2):308-314. doi:10.1016/j.spinee.2013.10.047 PubMedGoogle ScholarCrossref
38.
Lexchin  JR.  Implications of pharmaceutical industry funding on clinical research.   Ann Pharmacother. 2005;39(1):194-197. doi:10.1345/aph.1E224 PubMedGoogle ScholarCrossref
39.
Shamliyan  T, Kane  RL.  Clinical research involving children: registration, completeness, and publication.   Pediatrics. 2012;129(5):e1291-e1300. doi:10.1542/peds.2010-2847 PubMedGoogle ScholarCrossref
40.
Saa  C, Bunout  D, Hirsch  S.  Industry funding effect on positive results of probiotic use in the management of acute diarrhea: a systematized review.   Eur J Gastroenterol Hepatol. 2019;31(3):289-302. doi:10.1097/MEG.0000000000001322 PubMedGoogle ScholarCrossref
41.
Crockett  LK, Okoli  GN, Neilson  CJ, Rabbani  R, Abou-Setta  AM, Klassen  TP.  Publication of randomized clinical trials in pediatric research: a follow-up study.   JAMA Netw Open. 2018;1(1):e180156. doi:10.1001/jamanetworkopen.2018.0156 PubMedGoogle Scholar
42.
Klassen  TP, Wiebe  N, Russell  K,  et al.  Abstracts of randomized controlled trials presented at the Society for Pediatric Research meeting: an example of publication bias.   Arch Pediatr Adolesc Med. 2002;156(5):474-479. doi:10.1001/archpedi.156.5.474 PubMedGoogle ScholarCrossref
43.
Pica  N, Bourgeois  F.  Discontinuation and nonpublication of randomized clinical trials conducted in children.   Pediatrics. 2016;138(3):e20160223. doi:10.1542/peds.2016-0223 PubMedGoogle Scholar
44.
Turner  EH, Matthews  AM, Linardatos  E, Tell  RA, Rosenthal  R.  Selective publication of antidepressant trials and its influence on apparent efficacy.   N Engl J Med. 2008;358(3):252-260. doi:10.1056/NEJMsa065779 PubMedGoogle ScholarCrossref
45.
Hoffmann  DE, Fraser  CM, Palumbo  F, Ravel  J, Rowthorn  V, Schwartz  J.  Probiotics: achieving a better regulatory fit.   Food Drug Law J. 2014;69(2):237-272, ii.PubMedGoogle Scholar
46.
Lau  J, Ioannidis  JP, Terrin  N, Schmid  CH, Olkin  I.  The case of the misleading funnel plot.   BMJ. 2006;333(7568):597-600. doi:10.1136/bmj.333.7568.597 PubMedGoogle ScholarCrossref
47.
Pham  B, Platt  R, McAuley  L, Klassen  TP, Moher  D.  Is there a “best” way to detect and minimize publication bias? an empirical evaluation.   Eval Health Prof. 2001;24(2):109-125. doi:10.1177/01632780122034830 PubMedGoogle Scholar
48.
Hayashino  Y, Noguchi  Y, Fukui  T.  Systematic evaluation and comparison of statistical tests for publication bias.   J Epidemiol. 2005;15(6):235-243. doi:10.2188/jea.15.235 PubMedGoogle ScholarCrossref
49.
Sterne  JA, Gavaghan  D, Egger  M.  Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature.   J Clin Epidemiol. 2000;53(11):1119-1129. doi:10.1016/S0895-4356(00)00242-0 PubMedGoogle ScholarCrossref
50.
Ioannidis  JP, Trikalinos  TA.  The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey.   CMAJ. 2007;176(8):1091-1096. doi:10.1503/cmaj.060410 PubMedGoogle ScholarCrossref
51.
Scherer  RW, Saldanha  IJ.  How should systematic reviewers handle conference abstracts? a view from the trenches.   Syst Rev. 2019;8(1):264. doi:10.1186/s13643-019-1188-0 PubMedGoogle ScholarCrossref
Original Investigation
Pediatrics
October 8, 2021

Comparison of Publication of Pediatric Probiotic vs Antibiotic Trials Registered on ClinicalTrials.gov

Author Affiliations
  • 1Department of Pediatrics, Alberta Children’s Hospital, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada
  • 2Section of Pediatric Emergency Medicine, Department of Pediatrics, Alberta Children’s Hospital, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada
  • 3Department of Community Health Sciences and the Health Sciences Library, University of Calgary, Calgary, Alberta, Canada
  • 4Sections of Pediatric Emergency Medicine and Gastroenterology, Department of Pediatrics, Alberta Children’s Hospital Research Institute, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada
JAMA Netw Open. 2021;4(10):e2125236. doi:10.1001/jamanetworkopen.2021.25236
Key Points

Question  Do registered trials that evaluate the use of probiotics and routinely used antibiotics in children differ in their publication rates?

Findings  This cross-sectional analysis of 265 unique probiotic trials and 136 unique antibiotic trials in the ClinicalTrials.gov registry found that among registered trials conducted in children, probiotic trials are less likely to be published than antibiotic trials after adjustment for funding source, presence of blinding, and study purpose. No other evaluated study characteristics were independently associated with publication status in the regression model.

Meaning  These findings suggest that pediatric probiotic trials may be more prone to selective publication compared with pediatric antibiotic trials; therefore, meta-analyses and systematic reviews of probiotics must be interpreted cautiously.

Abstract

Importance  The published evidence in support of probiotic use is conflicting, which may be a result of selective publication of probiotic trials.

Objectives  To compare the proportion of registered trials that evaluate pediatric probiotics vs those that evaluate antibiotics that are published and to identify study-related factors associated with publication status.

Design, Setting, and Participants  This cross-sectional study evaluated eligible trials registered in ClinicalTrials.gov, an online clinical trials registry, from July 1, 2005, to June 30, 2016. Eligible studies included participants younger than 18 years, evaluated a probiotic or 1 of the 5 most commonly prescribed antibiotics in children and adolescents, and randomized study participants. All searches were updated and finalized as of September 9, 2020.

Exposures  Probiotic or antibiotic.

Main Outcomes and Measures  The primary outcome was study publication status. In addition, exposure status (probiotic vs antibiotic), trial result, and funding source were assessed for independent association with publication status. Whether study design elements, publication journal impact factor, and the interval from study completion to publication differed by exposure status were also evaluated.

Results  A total of 401 unique trials (265 probiotic and 136 antibiotic) met eligibility criteria. A greater proportion of antibiotic compared with probiotic studies were published (83 [61.0%] vs 119 [44.9%]; difference, 16.1% [95% CI, 5.8%-25.9%]). After adjustment for funding source, blinding, and purpose, studies evaluating an antibiotic were more likely to be published (odds ratio, 2.1 [95% CI, 1.3-3.4]). No other covariates included in the model were independently associated with publication status. Antibiotic trials, compared with probiotic trials, were more likely to have a therapeutic purpose (114 [83.8%] vs 117 [44.2%]; difference, 39.6% [95% CI, 31.1%-48.3%]) and to be multicenter (46 [33.8%] vs 46 [17.4%]; difference, 16.5% [95% CI, 7.5%-25.7%]). The median impact factor of the journals in which the studies were published was higher for the antibiotic trials (7.2 [IQR, 2.8-20.5] vs 3.0 [IQR, 2.3-4.2]; P < .001). The median number of days to publication did not differ between the probiotic and antibiotic trials (683 [IQR, 441-1036] vs 801 [IQR, 550-1183]; P = .24).

Conclusions and Relevance  The findings of this cross-sectional study suggest that probiotic studies are less likely to be published than antibiotic trials. No other study characteristics were associated with publication status. This finding raises concerns regarding the results of meta-analyses of probiotic trials.

Introduction

Probiotics represent a rapidly growing commercial industry valued at approximately US $15 billion in 2013.1 With growth estimated to be 7% per annum,2 there has been a concerted effort to document clinical effectiveness1,3,4 through the conduct of randomized clinical trials that are summarized in numerous meta-analyses.5-10 The latter inform guideline recommendations,11,12 which often support probiotic use with the caveat that the evidence is weak owing to the inclusion of studies with a high risk of bias.13-15 Although many early studies that influence the conclusions of meta-analyses reported benefits associated with probiotic use, recent multicenter trials often contradict these results,2,4,9,16,17 and they are beginning to inform the conclusions of reviews. For example, the 2010 Cochrane review of probiotic use in acute infectious diarrhea (63 studies) concluded that probiotics “have clear beneficial effects in shortening the duration and reducing stool frequency in acute infectious diarrhoea.”18(p2) However, the 2020 update (82 studies) states that “probiotics probably make little or no difference to the number of people who have diarrhoea lasting 48 hours or longer, and we are uncertain whether probiotics reduce the duration of diarrhoea.”19(p2)

Publication bias occurs when the direction or strength of a finding influences the likelihood, timing, language, and journal of publication.20 This phenomenon may explain the discordant conclusions of probiotic meta-analyses and the results of recent large, multicenter clinical trials.21 Because 25% to 50% of registered trials remain unpublished after study completion, meta-analyses may reach incorrect conclusions if the included data misrepresent the effects, benefits, and risks of the intervention.22,23 Publication bias is a particular concern in fields where industry-funded studies are common because they are associated with study discontinuation, nonpublication, and more favorable reporting of results and conclusions.24,25

The International Committee of Medical Journal Editors has embraced the use of registries, such as ClinicalTrials.gov, and since 2005 have required trial registration before participant enrollment as a prerequisite for publication.26 We therefore used this data source to compare the proportion of registered trials that are published between those evaluating a probiotic relative to those evaluating an antibiotic. Our secondary objective was to determine whether exposure status (ie, probiotic or antibiotic), trial result, or funding source were independently associated with publication status and whether study design elements, journal impact factor, and interval from study completion to publication differed by exposure status. We hypothesized that a lower proportion of registered probiotic trials are published compared with antibiotic trials.

Methods
Study Design and Setting

This cross-sectional study used publicly available aggregate data; thus, institutional review board approval was not required. The study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.

The study included trials registered in ClinicalTrials.gov with start dates from July 1, 2005, to June 30, 2016. Established in 1999, ClinicalTrials.gov is an online, publicly available registry that includes details on interventions, outcomes, results, and funding sources. The start date was selected to correspond with the 2005 International Committee of Medical Journal Editors requirement that trials be registered in advance of publication.27 The end date was chosen to allow for a sufficient interval between registration and study publication. Eligible studies met the following criteria: (1) the included participants were younger than 18 years (identified as children in the database); (2) the participants were randomized to at least 2 alternate interventions; and (3) the study evaluated a probiotic or 1 of the 5 most commonly prescribed antibiotics in children (eTable 1 in the Supplement).28,29 Antibiotics were selected to serve as the referent standard because these medications are no longer patent protected, and the financial implications of trial results would be minimal for industry stakeholders.

The registry search (eTable 2 in the Supplement) was undertaken with the assistance of a medical librarian (D.L.L.) using search terms for probiotics (Lactobacillus or probiotic or Saccharomyces or Enterococcus or Streptococcus or Acidophilus) and antibiotics (azithromycin or amoxicillin–clavulanic acid or amoxicillin or cefdinir or cephalexin). These terms were exploded to include synonyms through the ClinicalTrials.gov advanced search function (eTables 1 and 3 in the Supplement). Our search was limited to interventional trials. We did not apply any study purpose, language, result, or recruitment status restrictions. After the initial search, data were downloaded from ClinicalTrials.gov via comma-separated values export for review.

Eligibility Screening

All searches were updated and finalized as of September 14, 2020. Two authors (M.R. and K.L.) independently performed the search and evaluated studies for eligibility. After completion of independent screening and removal of duplicates, disagreements were resolved via consultation with a third reviewer (S.B.F.).

Data Extraction and Definitions

Two reviewers from a group of 3 (M.R., K.L., or N.L.) extracted data independently into a standardized database that was piloted with the first 10 eligible studies and subsequently refined before completing extraction for the remaining studies. If discrepancies occurred, the 2 reviewers attempted to resolve them; if discrepancies remained, a third reviewer (S.B.F.) was consulted.

Study completion was the date when participants were no longer being examined or treated (ie, final visit had occurred), as stated in the ClinicalTrials.gov registry. When this data field was incomplete, the estimated date of completion was extracted. Study duration was the interval from study start to completion dates. Data extractors classified the study purpose based on the following predefined criteria: therapeutic studies used an investigational agent to improve a specified medical condition; prophylactic studies evaluated investigational agent use for preventative purposes; safety/tolerability studies determined adverse effects associated with the investigational agent; and basic science studies were designed to assess pharmacokinetic properties. Data extractors classified the comparison group as placebo, standard care if the study described comparison to standard of care treatment, separate intervention if the comparison was another therapeutic agent, or alternate dosing if the trial compared alternative methods of administration, timing, or dosing of the intervention.

Blinding was present if any level of masking was reported on ClinicalTrials.gov. Trials are categorized by the registry into the following population age groups: premature infants (gestational age <36 weeks), term infants (aged 0-15 months), all ages (15 months to 18 years), and children and adults (includes participants older than 18 years). We categorized funding source according to the study sponsor disclosed on ClinicalTrials.gov. If any industry funding was disclosed, the study was categorized as industry funded.

Published trials had the following additional data extracted: whether findings for all primary outcomes were reported, direction of results, date, and journal of publication. We classified a study finding as positive if the results of the primary outcome, as stated in ClinicalTrials.gov, were statistically significant (ie, based on reported P values and/or 95% CIs).30 For noninferiority trials, if the investigational agent was found to be noninferior, the results were classified as positive. If there were multiple primary outcomes and the results were not uniformly positive or negative, these studies were classified as mixed. The publication interval was the difference between the study completion date (actual or estimated) and the date of publication. Journal impact factor was determined using the 2019 edition of Journal Citation Reports (Clarivate Analytics).31

Outcomes
Primary Outcome

Publication status, the primary outcome, was assessed in accordance with previous work in the field.23 First, the ClinicalTrials.gov registry was examined to identify whether an article citation was provided. If a citation was not identified, a PubMed search was performed using the ClinicalTrials.gov NCT number. If a publication was not located, PubMed was subsequently searched using the name of the principal investigator and other identifying descriptors (eg, population, intervention, and study outcomes).30 If a matching publication was not identified, the strategy was repeated using Google Scholar. All searches were updated and finalized as of September 9, 2020.

Secondary Objectives

We sought to determine whether study intervention (ie, probiotic vs antibiotic), outcome (ie, positive vs negative vs mixed), or funding source (ie, industry vs nonindustry) were independently associated with publication status. In addition, we aimed to determine whether study design elements (ie, blinding, study purpose, multicenter vs single-center), impact factor of the journal in which the study was published, and the interval from study completion to publication differed by study intervention type. These factors were selected as reflections of publication quality.32

Sample Size

Sample size calculations were performed using the Russ Lenth Java applet for determining power and sample size.33 We assumed a worst-case scenario and estimated that 50% of registered probiotic trials would be published. Based on a survey of 9 colleagues (3 pediatric emergency medicine physicians; 3 pediatric infectious disease specialists; and 3 gastroenterologists), the median clinically meaningful difference in the proportion of studies published was set at 15%. Using a 2-sided α of .05, a minimum of 135 studies were required in each cohort to reject the null hypothesis with 80% power.

Statistical Analysis

We report trial characteristics by exposure status (ie, probiotic or antibiotic) using descriptive statistics. Between-group comparisons for study design elements were performed using the Pearson χ2 test or the Fisher exact test if the number of studies in any single category was less than 10. For nonnormally distributed data, we used the Mann-Whitney test to compare 2 independent samples. Interrater reliability regarding publication status was determined using the Cohen κ value.34

For the primary outcome of publication status, we compared antibiotic and probiotic trials using the Pearson χ2 test. For the secondary outcomes, we conducted an a priori planned multiple logistic regression analysis to assess the association between publication status (dependent variable) and the type of intervention (ie, probiotic vs antibiotic) including the following a priori identified covariates: funding source, blinding, and study purpose. We compared features of study design, outcomes, and funding source between groups using the Pearson χ2 test. We used the Mann-Whitney test to compare impact factor of the journal in which the study was published and time to publication for the 2 groups.

Statistical analyses were completed on October 16, 2020, using SPSS Statistics Subscription Build, version 1.0.0.1461 (IBM Corp). Statistical significance was set at 2-sided P < .05.

Results

We identified 466 unique probiotic and 209 unique antibiotic trials in ClinicalTrials.gov. After screening, 265 (56.9%) probiotic and 136 (65.1%) antibiotic records were retained for a total of 401 studies (Figure 1). Fifty-eight probiotic trials reported their status as withdrawn, terminated, or unknown compared with 18 antibiotic trials (21.9% vs 13.2%; difference, 8.7% [95% CI, 0.5%-15.8%]). Probiotic trials were of shorter duration (median, 537 [IQR, 306-973] days) compared with antibiotic trials (median, 1096 [IQR, 668-1642] days; P = .001). Probiotic trials were more likely to be placebo-controlled compared with antibiotic trials (205 [77.4%] vs 55 [40.4%]; difference, 36.9% [95% CI, 26.9%-46.1%]), whereas antibiotic trials were more likely to use a separate intervention (38 [27.9%] vs 18 [6.8%]; difference, 21.2% [95% CI, 13.4%-30.0%]) or dosing regimen (24 [17.6%] vs 3 [1.1%]; difference, 16.5% [95% CI, 10.6%-23.8%]) as comparison groups. Results were more commonly available in ClinicalTrials.gov for the antibiotic studies (39 [28.7%] vs 17 [6.4%]; difference, 22.3% [95% CI, 14.4%-30.7%]) (Table 1).

Primary Outcome

Interreviewer publication status classification agreement for antibiotic and probiotic studies was outstanding, with κ values of 0.94 (95% CI, 0.90-0.98) and 0.82 (95% CI, 0.75-0.89), respectively. A greater proportion of antibiotic studies were published compared with probiotic studies (83 [61.0%] vs 119 [44.9%]; difference, 16.1% [95% CI, 5.8%-25.9%]).

Secondary Outcomes

After adjustment for industry funding, blinding, and study purpose, studies evaluating an antibiotic were more likely to be published (odds ratio, 2.1 [95% CI, 1.3-3.4]) (Table 2). Of the 83 published antibiotic studies, 56 (67.5%) reported statistically significant results, compared with 58 of the 119 published probiotic trials (48.7%; difference, 18.7% [95% CI, 4.9%-31.4%]). The proportion of industry-funded trials reporting statistically significant benefits was higher for antibiotic trials (10 of 13 [76.9%]) compared with probiotic trials (15 of 34 [44.1%]; difference, 32.8% [95% CI, 1.1%-54.1%]). A greater proportion (38 of 119 [31.9%]) of published probiotic trials were industry funded compared with published antibiotic trials (10 of 83 [12.0%]; difference, 19.9% [95% CI, 8.3%-30.2%]).

Antibiotic trials, compared with probiotic trials, were more likely to be therapeutic (114 of 136 [83.8%] vs 117 of 265 [44.2%]; difference, 39.7% [95% CI, 30.4%-47.5%]) and multicenter (46 of 136 [33.8%] vs 46 of 265 [17.4%]; difference, 16.5% [95% CI, 7.5%-25.7%]) (Table 1). A greater percentage of published antibiotic trials were multicenter in design (28 of 79 [35.4%]) compared with published probiotic trials (22 of 104 [21.2%]; difference, 14.2% [95% CI, 1.2%-27.2%]) (Figure 2). The median impact factor of the journals in which the articles were published was lower for the probiotic trials than antibiotic trials (3.0 [IQR, 2.3-4.2] vs 7.2 [IQR, 2.8-20.5]; P < .001) (eTable 4 in the Supplement). The median time to publication did not differ between the probiotic and antibiotic trials (801 [IQR, 550-1183] vs 683 [IQR, 441-1036] days; P = .24).

Exploratory Analyses

Industry-funded trials were more likely to involve premature or term infants compared with non–industry-funded trials (45 of 106 [42.5%] vs 73 of 295 [24.7%]; difference, 17.7% [95% CI, 7.3%-28.3%]). Antibiotic trials with statistically significant results were more likely to be multicenter (24 of 58 [41.4%]) compared with statistically significant probiotic trial findings (7 of 48 [14.6%]; difference, 26.8% [95% CI, 9.6%-41.6%]). Although the proportion of antibiotic trials has not changed over time (through 2010, 67 of 136 [49.3%]; after 2010, 69 of 136 [50.7%]; difference, 1.2% [95% CI, −13.7% to 16.0%]), the proportion of registered probiotic trials that were published has increased (through 2010, 91 of 264 [34.5%]; after 2010, 173 of 265 [65.3%]; difference, 31.0% [95% CI, 18.5%-42.3%]).

Discussion

In this study, we determined that antibiotic trials registered on ClinicalTrials.gov from July 1, 2005, to June 30, 2016, were more than twice as likely to be published compared with probiotic trials registered during the same period. Although characteristics such as funding source, study purpose, and blinding differed between probiotic and antibiotic studies, none of these were independently associated with publication status. The impact factor of the journal in which the studies were published was higher for the antibiotic studies.

Earlier evaluations of publication bias in the probiotic literature identified selective publication using statistical methods such as funnel plots and Egger intercept tests.35,36 One study35 reported that studies with negative findings were less likely to be published. The presence of significant publication bias35,36 led to a cautious interpretation of results that was reflected in the conclusions. We are unaware of any studies that have assessed publication bias in a field of research (eg, probiotics) by comparing publications with those of another field (eg, antibiotics). Although industry funding has been reported to influence the likelihood of publication,37-39 the only prior evaluation of this concern in the probiotic literature identified no such association.40 In our study, although industry funding was significantly greater among probiotic studies, it was not independently associated with publication status.

In general, pediatric studies are less likely to be published compared with studies that focus on adults.39,41-43 A study conducted in 201239 that used a design similar to that of our study reported that only 29% of included pediatric trials resulted in publication. Furthermore, those authors found that industry-funded pediatric trials were less likely to be published compared with those funded by the National Institutes of Health.39 However, there does appear to be a trend toward increasing publication of pediatric studies. A review of abstracts accepted to the Pediatric Academic Societies’ annual meetings from 1992 to 1995 found that 41% of abstracts were unpublished as of the year 2000.42 When this analysis was repeated 15 years later, only 28% of studies presented at this same meeting from 2008 to 2011 remained unpublished as of 2017.41 Larger sample sizes and registration on ClinicalTrials.gov are 2 factors that may explain the higher publication proportions.

Although we determined that pediatric probiotic studies are less likely to be published compared with pediatric antibiotic studies, our results do not pinpoint the underlying explanation. Although others have demonstrated a bias toward the publication of beneficial results using the US Food and Drug Administration registry and results database,44 no such data are available for probiotic studies, which are rarely conducted with the intention of applying for marketing approval or a labeling change.45 Thus, it is possible that the unpublished probiotic studies represent negative trials and that the higher level of selective publication in this field renders the results of meta-analyses unreliable.2

Selective publication is possible in all fields of research, and this can lead to inaccurate estimates of the effectiveness of an intervention.44 Selective publication wastes limited resources and the contributions of study participants and hinders the advance of medical knowledge. By altering the assessment of efficacy, selective publication can lead to inappropriate medication use and recommendations. Attempts to study selective publication are hindered by the unavailability of data from unpublished trials.44 Although a variety of tests are available to detect selective reporting bias, none reliably detect or exclude the possibility of bias.46-50

Strengths and Limitations

The strengths of our study include the large number of registered trials included for analysis that provide adequate power for our analyses. We designed our study to provide a large interval between study registration and assessment of publication status to provide sufficient time for studies to be completed and published. Our protocol for determining publication status and high interobserver reliability ensure confidence in labeling studies as unpublished. Our study is unique in that it compares selective publication across types of medications. This approach enables us to conclude that publication bias may be a greater concern in probiotic studies compared with those evaluating antibiotics in children.

A limitation of our study is the inability to determine the direction of results of trials that were unpublished and unavailable in ClinicalTrials.gov. Although the inclusion of conference abstracts in systematic reviews is a controversial topic,51 we did not include conference abstract publication in the definition of our primary outcome, which was focused on peer-reviewed journal publication indexed in PubMed or Google Scholar. We only used 1 comparison group—pediatric antibiotic trials—and ideally, subsequent analyses will include comparisons with other drug classes to enable similar comparative analyses to place this concern in the proper context. We also cannot conclusively determine the reason for nonpublication.

Conclusions

The findings of this cross-sectional study suggest that registered pediatric probiotic trials are less likely to be published than registered pediatric antibiotic studies. Although further research is needed to establish the underlying study characteristics that explain this finding, our results raise a concern regarding the accuracy of meta-analyses that rely on published data to reach their conclusions. These results serve to highlight the importance of closing the loopholes and barriers that allow studies to be conducted without publishing their findings. This evidence should drive regulatory measures to ensure access to unpublished trial data and the provision of reasons for nonpublication.

Back to top
Article Information

Accepted for Publication: June 27, 2021.

Published: October 8, 2021. doi:10.1001/jamanetworkopen.2021.25236

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Riddell M et al. JAMA Network Open.

Corresponding Author: Stephen B. Freedman, MDCM, MSc, Sections of Pediatric Emergency Medicine and Gastroenterology, Department of Pediatrics, Alberta Children’s Hospital Research Institute, Cumming School of Medicine, University of Calgary, 28 Oki Dr NW, Calgary, AB T3B 6A8, Canada (stephen.freedman@albertahealthservices.ca).

Author Contributions: Dr Riddell had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Riddell, Funk, Lorenzetti, Freedman.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Riddell.

Critical revision of the manuscript for important intellectual content: Riddell, Lam, Funk, Lorenzetti, Freedman.

Statistical analysis: Riddell, Funk, Freedman.

Obtained funding: Lam, Freedman.

Administrative, technical, or material support: Freedman.

Supervision: Funk, Freedman.

Conflict of Interest Disclosures: None reported.

Funding/Support: The research reported in this publication was supported by the O’Brien Center Summer Studentship and by the Alberta Children’s Hospital Foundation Professorship in Child Health and Wellness.

Role of the Funder/Sponsor: The sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the views of the O’Brien Health Science Program or Alberta Health Services.

References
1.
Caselli  M, Cassol  F, Calò  G, Holton  J, Zuliani  G, Gasbarrini  A.  Actual concept of “probiotics”: is it more functional to science or business?   World J Gastroenterol. 2013;19(10):1527-1540. doi:10.3748/wjg.v19.i10.1527 PubMedGoogle ScholarCrossref
2.
Freedman  SB, Schnadower  D, Tarr  PI.  The probiotic conundrum: regulatory confusion, conflicting studies, and safety concerns.   JAMA. 2020;323(9):823-824. doi:10.1001/jama.2019.22268 PubMedGoogle ScholarCrossref
3.
Foligné  B, Daniel  C, Pot  B.  Probiotics from research to market: the possibilities, risks and challenges.   Curr Opin Microbiol. 2013;16(3):284-292. doi:10.1016/j.mib.2013.06.008 PubMedGoogle ScholarCrossref
4.
Day  RL, Harper  AJ, Woods  RM, Davies  OG, Heaney  LM.  Probiotics: current landscape and future horizons.   Future Sci OA. 2019;5(4):FSO391. doi:10.4155/fsoa-2019-0004 PubMedGoogle Scholar
5.
Martinelli  M, Banderali  G, Bobbio  M,  et al.  Probiotics’ efficacy in paediatric diseases: which is the evidence? a critical review on behalf of the Italian Society of Pediatrics.   Ital J Pediatr. 2020;46(1):104. doi:10.1186/s13052-020-00862-z PubMedGoogle ScholarCrossref
6.
Khalesi  S, Bellissimo  N, Vandelanotte  C, Williams  S, Stanley  D, Irwin  C.  A review of probiotic supplementation in healthy adults: helpful or hype?   Eur J Clin Nutr. 2019;73(1):24-37. doi:10.1038/s41430-018-0135-9 PubMedGoogle ScholarCrossref
7.
Islam  SU.  Clinical uses of probiotics.   Medicine (Baltimore). 2016;95(5):e2658. doi:10.1097/MD.0000000000002658 PubMedGoogle Scholar
8.
Ciorba  MA.  A gastroenterologist’s guide to probiotics.   Clin Gastroenterol Hepatol. 2012;10(9):960-968. doi:10.1016/j.cgh.2012.03.024 PubMedGoogle ScholarCrossref
9.
Freedman  SB, Williamson-Urquhart  S, Farion  KJ,  et al; PERC PROGUT Trial Group.  Multicenter trial of a combination probiotic for children with gastroenteritis.   N Engl J Med. 2018;379(21):2015-2026. doi:10.1056/NEJMoa1802597 PubMedGoogle ScholarCrossref
10.
Baldassarre  ME, Di Mauro  A, Tafuri  S,  et al.  Effectiveness and safety of a probiotic-mixture for the treatment of infantile colic: a double-blind, randomized, placebo-controlled clinical trial with fecal real-time PCR and NMR-based metabolomics analysis.   Nutrients. 2018;10(2):E195. doi:10.3390/nu10020195 PubMedGoogle Scholar
11.
Marchand  V.  Using probiotics in the paediatric population.   Paediatr Child Health. 2012;17(10):575-576. doi:10.1093/pch/17.10.575 PubMedGoogle ScholarCrossref
12.
Thomas  DW, Greer  FR; American Academy of Pediatrics Committee on Nutrition; American Academy of Pediatrics Section on Gastroenterology, Hepatology, and Nutrition.  Probiotics and prebiotics in pediatrics.   Pediatrics. 2010;126(6):1217-1231. doi:10.1542/peds.2010-2548 PubMedGoogle ScholarCrossref
13.
Huang  R, Hu  J.  Positive effect of probiotics on constipation in children: a systematic review and meta-analysis of six randomized controlled trials.   Front Cell Infect Microbiol. 2017;7:153. doi:10.3389/fcimb.2017.00153 PubMedGoogle ScholarCrossref
14.
Goldenberg  JZ, Yap  C, Lytvyn  L,  et al.  Probiotics for the prevention of Clostridium difficile–associated diarrhea in adults and children.   Cochrane Database Syst Rev. 2017;12:CD006095. doi:10.1002/14651858.CD006095.pub4 PubMedGoogle Scholar
15.
Wojtyniak  K, Szajewska  H.  Systematic review: probiotics for functional constipation in children.   Eur J Pediatr. 2017;176(9):1155-1162. doi:10.1007/s00431-017-2972-2 PubMedGoogle ScholarCrossref
16.
Besselink  MG, van Santvoort  HC, Buskens  E,  et al; Dutch Acute Pancreatitis Study Group.  Probiotic prophylaxis in predicted severe acute pancreatitis: a randomised, double-blind, placebo-controlled trial.   Lancet. 2008;371(9613):651-659. doi:10.1016/S0140-6736(08)60207-X PubMedGoogle ScholarCrossref
17.
van den Nieuwboer  M, van de Burgwal  LHM, Claassen  E.  A quantitative key-opinion-leader analysis of innovation barriers in probiotic research and development: valorisation and improving the tech transfer cycle.   PharmaNutrition. 2016;4(1):9-18. doi:10.1016/j.phanu.2015.09.003Google ScholarCrossref
18.
Allen  SJ, Martinez  EG, Gregorio  GV, Dans  LF.  Probiotics for treating acute infectious diarrhoea.   Cochrane Database Syst Rev. 2010(11):CD003048. doi:10.1002/14651858.CD003048.pub3PubMedGoogle Scholar
19.
Collinson  S, Deans  A, Padua-Zamora  A,  et al.  Probiotics for treating acute infectious diarrhoea.   Cochrane Database Syst Rev. 2020;12(12):CD003048. doi:10.1002/14651858.CD003048.pub4PubMedGoogle Scholar
20.
Sterne  JA, Sutton  AJ, Ioannidis  JP,  et al.  Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials.   BMJ. 2011;343:d4002. doi:10.1136/bmj.d4002 PubMedGoogle ScholarCrossref
21.
Bellomo  R, Warrillow  SJ, Reade  MC.  Why we should be wary of single-center trials.   Crit Care Med. 2009;37(12):3114-3119. doi:10.1097/CCM.0b013e3181bc7bd5 PubMedGoogle ScholarCrossref
22.
Ekmekci  PE.  An increasing problem in publication ethics: publication bias and editors’ role in avoiding it.   Med Health Care Philos. 2017;20(2):171-178. doi:10.1007/s11019-017-9767-0 PubMedGoogle ScholarCrossref
23.
Ross  JS, Tse  T, Zarin  DA, Xu  H, Zhou  L, Krumholz  HM.  Publication of NIH funded trials registered in ClinicalTrials.gov: cross sectional analysis.   BMJ. 2012;344:d7292. doi:10.1136/bmj.d7292 PubMedGoogle ScholarCrossref
24.
Lundh  A, Lexchin  J, Mintzes  B, Schroll  JB, Bero  L.  Industry sponsorship and research outcome.   Cochrane Database Syst Rev. 2017;2:MR000033. doi:10.1002/14651858.MR000033.pub3PubMedGoogle Scholar
25.
Roddick  AJ, Chan  FTS, Stefaniak  JD, Zheng  SL.  Discontinuation and non-publication of clinical trials in cardiovascular medicine.   Int J Cardiol. 2017;244:309-315. doi:10.1016/j.ijcard.2017.06.020 PubMedGoogle ScholarCrossref
26.
DeAngelis  CD, Drazen  JM, Frizelle  FA,  et al; International Committee of Medical Journal Editors.  Clinical trial registration: a statement from the International Committee of Medical Journal Editors.   JAMA. 2004;292(11):1363-1364. doi:10.1001/jama.292.11.1363 PubMedGoogle ScholarCrossref
27.
Huser  V, Cimino  JJ.  Evaluating adherence to the International Committee of Medical Journal Editors’ policy of mandatory, timely clinical trial registration.   J Am Med Inform Assoc. 2013;20(e1):e169-e174. doi:10.1136/amiajnl-2012-001501 PubMedGoogle ScholarCrossref
28.
Hersh  AL, Shapiro  DJ, Pavia  AT, Shah  SS.  Antibiotic prescribing in ambulatory pediatrics in the United States.   Pediatrics. 2011;128(6):1053-1061. doi:10.1542/peds.2011-1337 PubMedGoogle ScholarCrossref
29.
Youngster  I, Avorn  J, Belleudi  V,  et al.  Antibiotic use in children: a cross-national analysis of 6 countries.   J Pediatr. 2017;182:239-244.e1. doi:10.1016/j.jpeds.2016.11.027 PubMedGoogle ScholarCrossref
30.
Bourgeois  FT, Murthy  S, Mandl  KD.  Outcome reporting among drug trials registered in ClinicalTrials.gov.   Ann Intern Med. 2010;153(3):158-166. doi:10.7326/0003-4819-153-3-201008030-00006 PubMedGoogle ScholarCrossref
31.
HeI H. Journal Citation Reports 2019 JCR, based on 2018 data (Clarivate Analytics, Thomson Reuters). 2020. Accessed September 7, 2020. https://mjl.clarivate.com/search-results
32.
Saha  S, Saint  S, Christakis  DA.  Impact factor: a valid measure of journal quality?   J Med Libr Assoc. 2003;91(1):42-46.PubMedGoogle Scholar
33.
Lenth  R. Java applets for power and sample size. 2018. Accessed October 8, 2020. https://stat.uiowa.edu/~rlenth/Power
34.
Landis  JR, Koch  GG.  The measurement of observer agreement for categorical data.   Biometrics. 1977;33(1):159-174. doi:10.2307/2529310 PubMedGoogle ScholarCrossref
35.
Videlock  EJ, Cremonini  F.  Meta-analysis: probiotics in antibiotic-associated diarrhoea.   Aliment Pharmacol Ther. 2012;35(12):1355-1369. doi:10.1111/j.1365-2036.2012.05104.x PubMedGoogle ScholarCrossref
36.
Ritchie  ML, Romanuk  TN.  A meta-analysis of probiotic efficacy for gastrointestinal diseases.   PLoS One. 2012;7(4):e34938. doi:10.1371/journal.pone.0034938 PubMedGoogle Scholar
37.
Amiri  AR, Kanesalingam  K, Cro  S, Casey  AT.  Does source of funding and conflict of interest influence the outcome and quality of spinal research?   Spine J. 2014;14(2):308-314. doi:10.1016/j.spinee.2013.10.047 PubMedGoogle ScholarCrossref
38.
Lexchin  JR.  Implications of pharmaceutical industry funding on clinical research.   Ann Pharmacother. 2005;39(1):194-197. doi:10.1345/aph.1E224 PubMedGoogle ScholarCrossref
39.
Shamliyan  T, Kane  RL.  Clinical research involving children: registration, completeness, and publication.   Pediatrics. 2012;129(5):e1291-e1300. doi:10.1542/peds.2010-2847 PubMedGoogle ScholarCrossref
40.
Saa  C, Bunout  D, Hirsch  S.  Industry funding effect on positive results of probiotic use in the management of acute diarrhea: a systematized review.   Eur J Gastroenterol Hepatol. 2019;31(3):289-302. doi:10.1097/MEG.0000000000001322 PubMedGoogle ScholarCrossref
41.
Crockett  LK, Okoli  GN, Neilson  CJ, Rabbani  R, Abou-Setta  AM, Klassen  TP.  Publication of randomized clinical trials in pediatric research: a follow-up study.   JAMA Netw Open. 2018;1(1):e180156. doi:10.1001/jamanetworkopen.2018.0156 PubMedGoogle Scholar
42.
Klassen  TP, Wiebe  N, Russell  K,  et al.  Abstracts of randomized controlled trials presented at the Society for Pediatric Research meeting: an example of publication bias.   Arch Pediatr Adolesc Med. 2002;156(5):474-479. doi:10.1001/archpedi.156.5.474 PubMedGoogle ScholarCrossref
43.
Pica  N, Bourgeois  F.  Discontinuation and nonpublication of randomized clinical trials conducted in children.   Pediatrics. 2016;138(3):e20160223. doi:10.1542/peds.2016-0223 PubMedGoogle Scholar
44.
Turner  EH, Matthews  AM, Linardatos  E, Tell  RA, Rosenthal  R.  Selective publication of antidepressant trials and its influence on apparent efficacy.   N Engl J Med. 2008;358(3):252-260. doi:10.1056/NEJMsa065779 PubMedGoogle ScholarCrossref
45.
Hoffmann  DE, Fraser  CM, Palumbo  F, Ravel  J, Rowthorn  V, Schwartz  J.  Probiotics: achieving a better regulatory fit.   Food Drug Law J. 2014;69(2):237-272, ii.PubMedGoogle Scholar
46.
Lau  J, Ioannidis  JP, Terrin  N, Schmid  CH, Olkin  I.  The case of the misleading funnel plot.   BMJ. 2006;333(7568):597-600. doi:10.1136/bmj.333.7568.597 PubMedGoogle ScholarCrossref
47.
Pham  B, Platt  R, McAuley  L, Klassen  TP, Moher  D.  Is there a “best” way to detect and minimize publication bias? an empirical evaluation.   Eval Health Prof. 2001;24(2):109-125. doi:10.1177/01632780122034830 PubMedGoogle Scholar
48.
Hayashino  Y, Noguchi  Y, Fukui  T.  Systematic evaluation and comparison of statistical tests for publication bias.   J Epidemiol. 2005;15(6):235-243. doi:10.2188/jea.15.235 PubMedGoogle ScholarCrossref
49.
Sterne  JA, Gavaghan  D, Egger  M.  Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature.   J Clin Epidemiol. 2000;53(11):1119-1129. doi:10.1016/S0895-4356(00)00242-0 PubMedGoogle ScholarCrossref
50.
Ioannidis  JP, Trikalinos  TA.  The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey.   CMAJ. 2007;176(8):1091-1096. doi:10.1503/cmaj.060410 PubMedGoogle ScholarCrossref
51.
Scherer  RW, Saldanha  IJ.  How should systematic reviewers handle conference abstracts? a view from the trenches.   Syst Rev. 2019;8(1):264. doi:10.1186/s13643-019-1188-0 PubMedGoogle ScholarCrossref
×