Factors Influencing the Publication of Randomized Controlled Trials in Child Health Research

Background  Publication bias threatens the validity of clinical decisions. The root causes are relatively unknown, and there is limited investigation in child research literature.

Objectives  To identify factors associated with subsequent nonpublication of abstracts presented at the Society for Pediatric Research meetings, and to determine the relative importance of the reasons identified for nonpublication.

Design  A cross-sectional survey was used to ask researchers about their reasons for the selective publication of randomized controlled trials (RCTs). The authors of 393 RCTs presented at the Society for Pediatric Research meetings from 1992 to 1995 were surveyed. A modified Total Design Method for mail surveys was used, with a reminder sent to all potential respondents 1 week after the initial mailing and full mailings sent to nonrespondents at 3 and 10 weeks following the initial mailing.

Results  One hundred sixty-six (45%) completed surveys were returned, and 119 (72%) abstracts were published as full manuscripts. Factors significantly associated with nonpublication identified through multiple logistic regression were the respondent’s report of scientific merit and significance of results. Of the 47 studies that were not published, only 8 (17%) had been submitted for publication. Authors of unpublished studies identified the following as important reasons for not publishing: not enough time (56.4 responded important or very important); trouble with coauthors (28.9); and journal unlikely to accept (26.3).

Conclusions  Of the RCTs presented and not subsequently published, the majority (83%) were never submitted for publication. The most common reason cited by authors for nonpublication was lack of time.

Publication bias is the selective publication of studies based on the direction and strength of the study results.¹ Consequently, smaller studies with nonsignificant or indefinite results are often subject to this bias. Publication bias threatens the validity of clinical decisions to the extent that these are based on the results of published literature to the exclusion of gray (ie, unpublished) literature. These decisions are vitally important, as they have the potential to affect financial, quality of life, and life and death decisions made on a daily basis. The existence of publication bias has been clearly demonstrated in many clinical areas.² Identification of the root causes may lead to possible solutions and subsequently increase our confidence in making unbiased decisions that will ultimately improve quality of care.

Reasons for nonpublication may rest with the editors, authors, the study itself, or other external factors. Previous research^3-9 suggests that the bias is rooted in the actions of the investigator rather than the actions of the editor. The principal reason for nonpublication that is consistently reported is lack of significant findings.^3-5,7,9-12 Other major reasons cited in the literature include disinterest on the part of the authors,^3,9,13 lack of time,^8,9,14,15 unimportant results,^5,7 and “coinvestigator or other operational problems.”^9,13 There is mixed evidence linking nonpublication to a study’s methodological quality,^{5,7,8,10,12,16} sample size,^{3-5,10,12,17,18} and other design features such as the number of data collection sites, presence of a control group, and study type.^4,7,8,13 Characteristics of the principal investigator, such as sex or academic rank, do not appear to play a role⁹; however, there is some suggestion that prior publication by the investigator is associated with subsequent publication.^8,19 Publication has also been linked to external funding.^4,7,13 Other potential factors external to the investigator and the study design include acceptance for presentation at a scientific meeting,^8,10,11 the type of presentation,^17,20 and the country where the study was conducted or reviewed.^11,21

The purpose of this study was to build on the work already completed in this area, to further illuminate the reasons behind publication bias with an exclusive focus on child health research. The specific objectives were to identify factors associated with subsequent publication and nonpublication of abstracts that appeared in Pediatric Research between 1992 and 1995, and to determine the relative importance of the reasons identified for nonpublication.

The study population consisted of 1 of the authors of each of the 393 abstracts of randomized controlled trials (RCTs) presented at the Society for Pediatric Research (SPR) meetings from 1992 to 1995. As part of an earlier study, abstracts of RCTs were identified by hand-searching the proceedings from the American Pediatric Society–SPR from 1992 to 1995, inclusive.¹² Abstracts were included if they reported phase III RCTs and measured pediatric outcomes; studies reporting outcomes on pregnant women were excluded. Abstracts were not included in the database if they only reported outcomes of nonrandomized treatment arms.

A questionnaire was designed specifically for the purposes of this study (available on request). The majority of the questions were based on factors that had been identified in the literature as potentially associated with nonpublication. These factors included statistical significance of the primary outcome as reported by the responding author (response categories were “statistically significant in favor of the test treatment,” statistically significant in favor of the control group, results favored the test treatment but did not meet statistical significance, results favored the control group but did not meet statistical significance, neither group was favored and results were not statistically significant, and other); whether the results were what the responding author had expected (yes or no); scientific merit of the study as rated by the responding author (poor, neutral, fair, good, excellent); clinical importance of the study and its findings as rated by the responding author (not important, neutral, somewhat important, important, very important); number of prior publications by the responding author; source of funding (no funding, governmental granting agency, private industry other than pharmaceutical company, pharmaceutical company, other); and reasons for nonpublication (see Table 1 for response categories; authors were also given the option to add additional reasons and comments). Other basic questions were asked regarding the responding author (eg, age at time of the abstract, sex, academic rank at the time of the study) and the study (eg, whether the abstract was published as a full manuscript, where the manuscript had been submitted and published, the year of publication, the number of publications from the same trial, and, if not published, whether it had been submitted for publication).

The questionnaire was pilot tested for content and face validity among a convenience sample of 8 researchers. These individuals were asked to complete the questionnaire with respect to their own research that was reported elsewhere between 1992 and 1995. They were also asked to provide feedback regarding the format and appearance of the questionnaire, time to complete, and the comprehensibility and applicability of the questions. Based on their responses, modifications were made to the questionnaire prior to starting the study.

Contact information in the form of either postal or e-mail addresses was sought for the first author mentioned in the abstract. If no information was found for the first author, an address for the next author was sought, and the next author after that if the situation repeated itself. Those authors for whom we found both a postal mailing address and an e-mail address were randomized to receive the survey by 1 of these 2 methods. The results of the randomized trial comparing responses for postal vs e-mail are the subject of another article.²² If only 1 type of mailing address was found, the survey was sent using this method of mailing.

The Total Design Method (TDM) for mail surveys was used as a guideline for the mailing procedure.²³ One week after the initial questionnaire was mailed, a reminder was sent. Three weeks after the initial mailing, a second complete package was sent to those individuals who had not yet responded. Ten weeks after the initial mailing, a third and final complete package was sent to those individuals who had still not responded. No incentives were offered for completing the surveys.

Data from the completed questionnaires were coded and entered into Microsoft Access 2000 (Microsoft Corporation, Remond, Wash). Data entry was checked for accuracy by a second researcher. All data analyses were conducted using SPSS for Windows Version 11.0.1 (SPSS Inc, Chicago, Ill). For descriptive purposes, variables were analyzed using medians and interquartile ranges (IQRs) for nonparametric data, means and SDs for normal continuous data, and percentages and 95% confidence intervals (CIs) for dichotomous/categorical data. Pearson χ² tests were used to measure the association between different variables and publication status. Multiple logistic regression was used to simultaneously evaluate factors associated with publication status.

The study was reviewed and approved by the University of Alberta Health Research Ethics Board.

One hundred sixty-six (45.5%) surveys were completed (refusals are included in the denominator but not in the numerator; 28 surveys were returned as undeliverable and were excluded from the denominator). The majority of respondents were men (77.7%) between the ages of 30 and 49 years (79.5%). According to the responding authors, 71.7% of abstracts were subsequently published as full manuscripts. Among the trials that were subsequently published, the median number of submissions prior to acceptance for publication was 1.0 (range, 1-7). The median number of publications per study was 1.0 (IQR, 1-2; range, 0-10); 26.1% of the studies generated more than 1 publication. The median number of years to publication was 2.0 (IQR, 1-2; range, 1-7). The median impact factor of the journals in which the trials were published was 3.536 (IQR, 2.089-3.708; range, 0.091-29.065). The journals in which the trials were most frequently published were Pediatrics (21.8%), Journal of Pediatrics (13.4%), New England Journal of Medicine (7.6%), Pediatric Pulmonology (5.9%), Journal of Perinatology (5.0%), Archives of Diseases of Childhood (4.2%), and Archives of Pediatrics and Adolescent Medicine (4.2%).

Of the 47 studies that were not published, only 8 (17%) had been submitted for publication. Respondents were asked to rate the importance of possible reasons for nonpublication (Table 1). The reasons most often cited as important or somewhat important were not enough time (56.4%), too much trouble with coauthors (28.9%), authors thought that a journal was unlikely to accept the study for publication (26.3%), and results were not statistically significant (23.7%). Respondents were also asked to identify the most important reason for nonpublication:too much trouble with coauthors (32.3%), followed by not enough time (25.8%).

Variables that were independently associated with subsequent publication are listed in Table 2. The following characteristics of respondents were tested and not found to be associated with subsequent publication: academic rank, number of prior publications, age, and sex.

Variables significantly associated with publication identified through multiple logistic regression were respondents’ reports of scientific merit and significant results. Studies for which the respondents described the primary outcome as statistically significant were more likely to be published (adjusted odds ratio [OR], 5.0; 95% CI, 2.1-12). Studies with a higher rating of scientific merit (as judged by the respondent) were more likely to be published. This included good vs fair, neutral, or poor (adjusted OR, 4.0; 95% CI, 1.5-11) and excellent vs fair, neutral, or poor (adjusted OR, 19.0; 95% CI, 4.2-83).

We identified factors associated with publication from a large sample of RCTs presented at the SPR's annual meetings between 1992 and 1995. Our study found that the most important factors associated with subsequent publication are the authors’ reports of the scientific merits of the studies and the significance of the results. Previous research in other areas has found that the rating of scientific importance by the authors is associated with publication.^5,7 We examined perceived scientific merit (methodological quality) and clinical importance (impact of results on clinical practice) as separate variables. While both were found to be significantly associated with subsequent publication, the association was stronger for perceived scientific merit. The finding that positive results are associated with publication is also consistent with previous research.^{7,10,14,16,24} In contrast to these findings, an evaluation of publication bias in gastroenterological research found no significant association between direction of study results and subsequent publication; however, statistically significant results did influence the likelihood of publication in high-impact journals.⁸

Consistent with previous research,^7,8,13 we found that the reasons for nonpublication were often related to authors’ actions and perceptions. The most common reasons put forward by authors for nonpublication were lack of time, too much trouble with coauthors, authors thought the journal was unlikely to accept the study for publication, and results were not statistically significant. A recent study provided evidence to discredit the idea that studies with positive results are more likely to be published: Olson et al²⁵ found no significant effect of the direction of study results on publication for manuscripts of controlled trials submitted to JAMA from February 1996 to August 1999.

Preventing publication bias is critical for several reasons, including ethical responsibility; accountability to funding sources; ensuring availability of information for other research, such as systematic reviews; and clinical decision-making. Many have supported the idea of a trials register to prevent publication bias, whereby all trials would be prospectively registered at their outset.^{1,7,13,16,26-28} This would eliminate the factors associated with nonpublication that occur at the end of the research project, such as lack of time, unimportant findings, or trouble with coinvestigators. The likelihood for success of such an initiative may be optimized if it is part of the ethics approval process.

The most important limitation of the present study is the response rate; the results must be interpreted in light of this. In a previous study of this sample,¹² we showed that the subsequent publication rate was approximately 60% compared with 72% in the present study. While there may have been a small degree of misclassification for various reasons, including inability to find the study in the databases searched,⁸ this is unlikely to account for the full discrepancy. The discrepancy suggests that authors who had published were more likely to respond to our survey; further analyses showed that the response rates were 38% vs 56% for unpublished and published studies, respectively. However, in identifying reasons for nonpublication, there is no reason to believe that the nonpublishing authors who responded would be different from those who had not. A second limitation is that the data were self-reported and may be susceptible to recall or reporting bias.³ For instance, the authors that subsequently published (or did not publish) may perceive their study and its findings differently based on their experience during manuscript submission and publication.

Article

Publication bias threatens the validity of clinical decisions. The reasons for nonpublication appear to be related to actions of the investigator rather than those of the editor. Trials registers have been widely supported to prevent bias associated with nonpublication. These study results imply that the most important factors associated with publication among a sample of abstracts presented at a large pediatric conference between 1992 and 1995 were scientific merit, as rated by the researcher, and significance of the findings. The reasons for nonpublication were often related to authors’ actions and perceptions. The most common reasons cited for nonpublication were lack of time, trouble with coauthors, journal unlikely to accept, and results not statistically significant.

Correspondence: Lisa Hartling, BScPT, MSc, Alberta Research Centre for Child Health Evidence, Department of Pediatrics, University of Alberta, Aberhart Center One, Room 9424, 11402 University Ave, Edmonton, Alberta, Canada T6G 2J3 (lisa.hartling@ualberta.ca).

Accepted for Publication: May 27, 2004.

Funding/Support: The Alberta Research Centre for Child Health Evidence is supported by an establishment grant from the Alberta Heritage Foundation for Medical Research.

Acknowledgment: We thank Jacqueline Jubinville and Belinda Allan for administrative support. We thank Natasha Wiebe for statistical consultation. We also thank the researchers who pilot tested the data collection instrument and the authors who responded to the survey.