[Skip to Content]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address 54.166.5.76. Please contact the publisher to request reinstatement.
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
Peer Review Congress
July 15, 1998

Publication Bias and Research on Passive SmokingComparison of Published and Unpublished Studies

Author Affiliations

From the Institute for Health Policy Studies (Ms Misakian and Dr Bero) and the Department of Clinical Pharmacy (Dr Bero), University of California, San Francisco.

JAMA. 1998;280(3):250-253. doi:10.1001/jama.280.3.250
Context.—

Context.— The results of reviews may be biased by delays in publication and failure to publish nonsignificant results.

Objective.— To determine the extent of unpublished results on the health effects of passive smoking and whether passive smoking studies with statistically nonsignificant results would have longer time to publication than those with statistically significant results.

Design.— Semistructured telephone interviews of principal investigators of published or unpublished studies funded between 1981 and 1995, identified by information obtained from 76 (85%) of 89 organizations contacted that potentially funded research on passive smoking.

Participants.— Seventy-eight investigators were eligible and could be located; 65 (83%) responded. They had conducted 61 studies of the health effects of passive smoke in humans or animals between 1981 and 1995 that met the criteria for the analysis of time to publication.

Main Outcome Measure.— Time to publication for published studies and statistical significance of results of published and unpublished studies.

Results.— Fourteen of the 61 studies were unpublished. Median time to publication was 5 years (95% confidence interval [CI], 4-7 years) for statistically nonsignificant studies and 3 years (95% CI, 3-5 years) for statistically significant studies (P=.004). Statistically significant results (P=.004), experimental study design (P=.01), study size less than or equal to 500 (P=.01), and animals as subjects (P=.03) were predictive of time to publication. When the studies with human participants were analyzed separately, only statistically significant data were predictive of publication (P=.007). Multivariate analysis of all studies indicated that statistical significance (P=.001) and study design (P=.01) were the only independent predictors of time to publication, while for the human studies only statistical significance was predictive of publication (P=.007).

Conclusion.— There is a publication delay for passive smoking studies with nonsignificant results compared with those with significant results.

A GREATER proportion of studies with statistically significant results are published than those with nonsignificant results.1,2 Publication bias is the tendency for investigators to submit manuscripts and for editors to accept them based on the statistical significance (in either direction) of the study results.35 Publication bias can limit access to important data and skew the results of reviews.

Some have argued that reviews concluding that passive smoking is harmful69 are invalid because publication bias exists.1016 Previously, we found that 20% of peer-reviewed articles examining the health effects of passive smoking had statistically nonsignificant results.17 While our analysis of the published literature on passive smoking suggests that statistically nonsignificant results are published, it tells us little about the statistical significance of unpublished results. Therefore, in this study, we interviewed the principal investigators of passive smoking studies to determine the extent of unpublished results in this area.

Even when clinical studies with statistically nonsignificant findings are published, there is a time lag in their publication compared with studies with statistically significant findings.1820 Thus, we hypothesized that passive smoking studies with statistically nonsignificant results would take longer to be published than studies with statistically significant results. We also hypothesized that, as for clinical trials, study design, sample size, or funding source might influence the time to publication.

METHODS
Identification of Passive Smoking Research Studies

Included studies were as follows: (1) in vivo studies of passive smoke exposure in humans or animals; (2) studies of the effects of passive smoking on disease, disability, or comfort; (3) original research studies; and (4) studies funded between 1981 (the year the first study on the health effects of passive smoking was published21) and 1995 for which data collection had begun by the time of the interview. Studies that analyzed passive smoking as a covariant, reviews, policy analyses, or biochemical mechanism studies were excluded.

To identify studies, we contacted 89 organizations that potentially provided extramural or intramural funding for passive smoking research. We identified these funding agencies by examining the acknowledgment sections of published papers on passive smoking,22,23 contacting experts in tobacco research, and conducting a search of a database of foundations maintained by the Foundation Center, New York, NY.

We received information from 76 (85%) of 89 organizations contacted. The response rates were 100% (12/12) for governmental agencies, 88% (60/68) for privately funded organizations, and 56% (5/9) for tobacco industry–affiliated organizations.

Interviews

This study (CHR H2758-08585-05) was approved by the Committee on Human Research at the University of California, San Francisco. Principal investigators of eligible studies were invited by letter to participate in a semistructured telephone interview. During a preinterview, investigators who agreed to participate answered a short series of questions to confirm that their study met the inclusion criteria. The investigator was also asked whether he/she had been the principal investigator for any other studies of the health effects of passive smoking. These additional studies were included in the interview if they met our inclusion criteria. An appointment for the full telephone interview was scheduled.

Shortly before the scheduled interview, investigators were faxed a confirmation letter that included a list of materials to have on hand during the interview (a list of publications, the grant proposal, and any progress reports, unpublished data summaries, talks, or abstracts). All interviews were conducted by one of the authors (A.L.M.) between January 1996 and April 1997.

To determine the extent of unpublished results, we asked investigators to identify all of their published and unpublished results on the health effects of passive smoking. We also asked investigators about their publications, study design characteristics, and reason(s) for unpublished results based on an instrument used by Dickersin et al.1,24

Analysis of Time to Publication

To test our hypothesis that the statistical significance of results, study design characteristics, and funding sources are associated with time to publication, we classified studies as "published" or "unpublished" and collected information on the variables listed below during the interviews. Interview data were supplemented with data from the granting agency or the first publication from the study.

Studies were excluded from the analysis of time to publication if (1) data analysis had not yet begun as the investigators were not able to answer our questions about the statistical significance of the data or final sample size; (2) the publications associated with the study were published before the start of the grant, as these publications could not be attributed to the funded project; or (3) the publications associated with the study contained no statistical analyses, as statistical significance could not be categorized. The remaining studies were classified as "published" if any of the results had been published in a peer-reviewed, non–peer-reviewed, or in-press publication. Studies were classified as "unpublished" if none of the results had been published or if results had been published only as abstracts, since investigators were unable to provide copies of their abstracts or the abstracts did not contain sufficient information.

Variables

Time to Publication.— Time to publication was calculated by subtracting the year funding began from the year of publication of the first original research article on the health effects of passive smoking resulting from the grant. Unpublished studies were censored at the year the interview was conducted. The funding start date was used to measure time to publication since the completion date was difficult to define.

Statistical Significance of Data.— The statistical significance of each study was classified as "statistically significant" (results indicate a statistically significant association between passive smoking and a harmful health effect); "statistically nonsignificant" (results do not indicate a statistically significant association); or "mixed" (multiple primary outcomes were measured and at least 1 was statistically significant). "Statistically significant" was defined as a P ≤.05, an odds ratio not overlapping 1, or other statistics typically characterized as significant. For unpublished studies, the categorization of statistical significance was based on the investigator's classification of his/her unpublished results. For published studies, the categorization was based on the significance of the results presented in the first original research publication as determined by 1 of the authors (A.L.M.) and a doctorate student in epidemiology.

Study Subjects.— Classified as "animals" or "humans."

Study Design.— A study was classified as "experimental" if a test regimen initiated by the investigator was administered to a treatment group, but not a control group, or as "observational" if the study was a cohort, case-control, or descriptive study.

Study Size.— Classified as "500 or less" or "more than 500" based on the distribution of sample sizes.

Health Outcome.— Classified as "respiratory effects," "lung cancer," "cardiovascular effects," "pregnancy outcome and/or birth defects," and "other" (includes studies with mixed outcomes).

Funding Source and Length.—The primary funding source was identified by the investigator and was categorized as "government," "private," "tobacco industry," or "other" and as "external" (investigator not affiliated with funding agency) or "internal." Length of funding was classified as "5 years or less" or "more than 5 years" based on the distribution of funded studies.

Statistical Analysis

The χ2 statistic was used to compare the characteristics of participants and refusals. Predictors of time to publication were analyzed by proportional hazards models.25 When there was evidence against the proportional hazards assumptions,26P values were obtained by the nonparametric Wilcoxon test.27 Multivariate models were built using stepwise addition of variables until no further additions provided a statistically significant improvement in prediction at P =.05.

RESULTS
Identification of Passive Smoking Research Studies

We identified 105 investigators eligible for inclusion. Four investigators could not be located, 21 were found to be ineligible after completing the preinterview, and 2 were eliminated after being interviewed because their studies had no results on the health effects of passive smoking. The response rate among the remaining 78 investigators was 83% (65/78). Characteristics of participants and refusals are shown in Table 1. Of the 13 investigators who chose not to participate, 5 cited a lack of time, 3 felt uncomfortable participating, and 5 did not state a reason.

Table 1.—Characteristics of Participants and Refusals*
Table 1.—Characteristics of Participants and Refusals*
Image description not available.

The 65 participants provided information on 84 studies. Sixty-six (79%) of these studies were identified based on the information from granting organizations and 18 (21%) were identified by the investigators.

Fifty-nine (70%) of the 84 studies had some unpublished results. The reasons stated most frequently for unpublished results were ongoing data collection or analysis (n=33 times), lack of time (n=26), and competing priorities (n=11), such as other recently funded studies, studies where passive smoking was a minor component, and career changes. Statistically nonsignificant results were cited as a reason for failure to publish for only 2 studies. One investigator stated that they chose a less prestigious journal to publish their statistically nonsignificant results. Four manuscripts (2 with statistically significant and 2 with statistically nonsignificant results) resulting from 3 studies were unpublished because they had been rejected from a journal. These manuscripts were being resubmitted.

Analysis of Time to Publication

Of the 84 identified studies, 23 were excluded from the analysis of time to publication because data analysis had not begun (n=19), the publications were published before the start of the grant (n=2), or the publications had no statistical analyses (n=2). Of the remaining 61 studies, 47 were classified as "published" and 14 as "unpublished."

Figure 1 shows that the survival curve plots for time to publication for all studies differed by the statistical significance of the study results (P=.004). The median time to publication was 5 years (95% confidence interval [CI], 4-7 years) for statistically nonsignificant studies (n=21) and 3 years (95% CI, 3-5 years) for statistically significant studies (n=33). Only 1 of the 7 studies with mixed results was published; the time to publication was 6 years.

Image description not available.
Proportion of unpublished studies by statistical significance of results.

As shown in Table 2, univariate analysis of all studies showed that statistically significant results, experimental study design, study size of 500 or less, and animals as subjects were predictive of time to publication. When the studies with human participants were analyzed separately (Table 2), only the statistical significance of the results was predictive of time to publication. Multivariate analysis of all studies indicated that statistical significance (P=.001) and study design (P=.01) were the only independent predictors of time to publication (P>.16 for all other variables). Multivariate analysis of the human studies showed that only statistical significance was predictive of time to publication (P=.007) (P>.15 for all other variables).

Table 2.—Comparison of Published and Unpublished Studies*
Table 2.—Comparison of Published and Unpublished Studies*
Image description not available.
COMMENT

Our results indicate that statistically significant studies are published sooner than statistically nonsignificant studies, as reported for clinical studies.19,28 This finding cannot be attributed to differences in the sample sizes, funding sources, or health outcomes measured. Our finding of a publication delay can be partially explained by the reasons that investigators cited for not publishing.

The delay in the publication of statistically nonsignificant results has implications for conducting reviews and meta-analyses. The time lag in publication argues for regular updating of reviews, as practiced by the Cochrane Collaboration,29 as statistical significance of a review could change over time. Furthermore, defining eligibility for inclusion in a review by the start date of a study could reduce bias in reviews by including unpublished results. Since 47 (77%) of the 61 studies we identified had at least some published results, researchers conducting reviews on passive smoking should be aware of these studies and seek data from them.

One limitation of our study is that we interviewed only English-speaking investigators funded by US organizations. Therefore, we may have underestimated the number of studies of the health effects of passive smoking. A second limitation is that we had to rely on principal investigators to tell us the statistical significance of their unpublished results. Finally, our ability to identify studies was limited by the inadequate record keeping and willingness to share information on the part of funding agencies or investigators. For example, only half of the tobacco-affiliated organizations responded to our multiple requests for information. Our difficulty with identifying unpublished studies supports the need to register prospectively observational studies, as has been suggested for clinical trials.30

References
1.
Dickersin K, Min YI. Publication bias: the problem that won't go away.  Ann N Y Acad Sci.1993;703:135-146. Discussion 146-8.
2.
Dickersin K. How important is publication bias? a synthesis of available data.  AIDS Educ Prev.1997;9(suppl A):15-21.
3.
Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H. Publication bias and clinical trials.  Control Clin Trials.1987;8:343-353.
4.
Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research.  Lancet.1991;337:867-872.
5.
Rennie D, Flanagin A. Publication bias: the triumph of hope over experience.  JAMA.1992;267:411-412.
6.
US Environmental Protection Agency.  Respiratory Health Effects of Passive Smoking: Lung Cancer and Other Disorders . Washington, DC: Indoor Air Division, Office of Atmospheric and Indoor Air Programs, Office of Air and Radiation, US Environmental Protection Agency; 1992. No. EPA/600/6-90/006F.
7.
US Department of Health and Human Services.  The Health Consequences of Involuntary Smoking: A Report of the Surgeon General . Washington, DC: US Dept of Health and Human Services; 1986:332. DHHS Publication (CDC) 87-8398.
8.
Law MR, Morris JK, Wald NJ. Environmental tobacco smoke exposure and ischaemic heart disease: an evaluation of the evidence.  BMJ.1997;315:973-980.
9.
Hackshaw AK, Law MR, Wald NJ. The accumulated evidence on lung cancer and environmental tobacco smoke.  BMJ.1997;315:980-988.
10.
Vandenbroucke JP. Passive smoking and lung cancer: a publication bias?  Br Med J (Clin Res Ed).1988;296:391-392.
11.
Mantel N. The passive smoking myth. In: Bieva CJ, Courtois Y, Govaerts M, eds. The Present and Future of Indoor Air Quality . Amsterdam, the Netherlands: Excerpta Medica; 1989:155-159.
12.
Kilpatrick SJ. The epidemiology of environmental tobacco smoke (ETS) and the weight of evidence argument.  Int Surg.1992;77:131-133.
13.
Sugita M, Kanamori M, Izuno T, Miyakawa M. Estimating a summarized odds ratio whilst eliminating publication bias in meta-analysis.  Jpn J Clin Oncol.1992;22:354-358.
14.
Armitage AK. Environmental tobacco smoke and coronary heart disease.  J Smoking-Related Dis.1993;4:27-36.
15.
Caldwell ADS. Give a dog-end a bad name.  J Smoking-Related Dis.1992;4:1-2.
16.
The Tobacco Institute.  Science does not support draft documents undergoing expert review [press release]. Washington, DC: The Tobacco Institute; 1990.
17.
Bero L, Glantz S, Rennie D. Publication bias and public health policy on environmental tobacco smoke.  JAMA.1994;272:133-136.
18.
Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects.  BMJ.1997;315:640-645.
19.
Ioannidis JPA. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials.  JAMA.1998;279:281-286.
20.
Jadad A, Rennie D. The randomized controlled trial gets a middle-aged checkup.  JAMA.1998;279:319-320.
21.
Hirayama T. Non-smoking wives of heavy smokers have a higher risk of lung cancer: a study from Japan.  Br Med J (Clin Res Ed).1981;282:183-185.
22.
Barnes D, Bero L. Scientific quality of original research articles on environmental tobacco smoke.  Tob Control.1997;6:19-26.
23.
Bero LA, Galbraith A, Rennie D. The publication of sponsored symposiums in medical journals.  N Engl J Med.1992;327:1135-1140.
24.
Dickersin K, Min Y-I, Meinert CL. Factors influencing publication of research results: follow-up of applications submitted to two institutional review boards.  JAMA.1992;267:374-378.
25.
Cox DR. Regression models and life tables (with discussion).  J R Stat Soc B.1972;34:187-220.
26.
Harrell FE, Lee KL. Verifying Assumptions of the Cox Proportional Hazards Model: Proceedings of the 11th Annual SAS User's Group International Conference. Cary, NC: SAS Institute Inc; 1986.
27.
Cox DR, Oakes D. Analysis of Survival Data . London, England: Chapman & Hall; 1984:123-128.
28.
Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects.  BMJ.1997;315:640-645.
29.
Bero LA, Rennie D. The Cochrane Collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care.  JAMA.1995;274:1935-1938.
30.
Simes RJ. Publication bias: the case for an international registry of clinical trials.  J Clin Oncol.1986;4:1529-1541.
×