From the Institute for Health Policy Studies (Ms Misakian and Dr Bero) and the Department of Clinical Pharmacy (Dr Bero), University of California, San Francisco.
Context.— The results of reviews may be biased by delays in publication and failure
to publish nonsignificant results.
Objective.— To determine the extent of unpublished results on the health effects
of passive smoking and whether passive smoking studies with statistically
nonsignificant results would have longer time to publication than those with
statistically significant results.
Design.— Semistructured telephone interviews of principal investigators of published
or unpublished studies funded between 1981 and 1995, identified by information
obtained from 76 (85%) of 89 organizations contacted that potentially funded
research on passive smoking.
Participants.— Seventy-eight investigators were eligible and could be located; 65 (83%)
responded. They had conducted 61 studies of the health effects of passive
smoke in humans or animals between 1981 and 1995 that met the criteria for
the analysis of time to publication.
Main Outcome Measure.— Time to publication for published studies and statistical significance
of results of published and unpublished studies.
Results.— Fourteen of the 61 studies were unpublished. Median time to publication
was 5 years (95% confidence interval [CI], 4-7 years) for statistically nonsignificant
studies and 3 years (95% CI, 3-5 years) for statistically significant studies
(P=.004). Statistically significant results (P=.004), experimental study design (P=.01), study size less than or equal to 500 (P=.01),
and animals as subjects (P=.03) were predictive of
time to publication. When the studies with human participants were analyzed
separately, only statistically significant data were predictive of publication
(P=.007). Multivariate analysis of all studies indicated
that statistical significance (P=.001) and study
design (P=.01) were the only independent predictors
of time to publication, while for the human studies only statistical significance
was predictive of publication (P=.007).
Conclusion.— There is a publication delay for passive smoking studies with nonsignificant
results compared with those with significant results.
A GREATER proportion of studies with statistically significant results
are published than those with nonsignificant results.1,2
Publication bias is the tendency for investigators to submit manuscripts and
for editors to accept them based on the statistical significance (in either
direction) of the study results.3- 5
Publication bias can limit access to important data and skew the results of
Some have argued that reviews concluding that passive smoking is harmful6- 9 are invalid
because publication bias exists.10- 16
Previously, we found that 20% of peer-reviewed articles examining the health
effects of passive smoking had statistically nonsignificant results.17 While our analysis of the published literature on
passive smoking suggests that statistically nonsignificant results are published,
it tells us little about the statistical significance of unpublished results.
Therefore, in this study, we interviewed the principal investigators of passive
smoking studies to determine the extent of unpublished results in this area.
Even when clinical studies with statistically nonsignificant findings
are published, there is a time lag in their publication compared with studies
with statistically significant findings.18- 20
Thus, we hypothesized that passive smoking studies with statistically nonsignificant
results would take longer to be published than studies with statistically
significant results. We also hypothesized that, as for clinical trials, study
design, sample size, or funding source might influence the time to publication.
Included studies were as follows: (1) in vivo studies of passive smoke
exposure in humans or animals; (2) studies of the effects of passive smoking
on disease, disability, or comfort; (3) original research studies; and (4)
studies funded between 1981 (the year the first study on the health effects
of passive smoking was published21) and 1995
for which data collection had begun by the time of the interview. Studies
that analyzed passive smoking as a covariant, reviews, policy analyses, or
biochemical mechanism studies were excluded.
To identify studies, we contacted 89 organizations that potentially
provided extramural or intramural funding for passive smoking research. We
identified these funding agencies by examining the acknowledgment sections
of published papers on passive smoking,22,23
contacting experts in tobacco research, and conducting a search of a database
of foundations maintained by the Foundation Center, New York, NY.
We received information from 76 (85%) of 89 organizations contacted.
The response rates were 100% (12/12) for governmental agencies, 88% (60/68)
for privately funded organizations, and 56% (5/9) for tobacco industry–affiliated
This study (CHR H2758-08585-05) was approved by the Committee on Human
Research at the University of California, San Francisco. Principal investigators
of eligible studies were invited by letter to participate in a semistructured
telephone interview. During a preinterview, investigators who agreed to participate
answered a short series of questions to confirm that their study met the inclusion
criteria. The investigator was also asked whether he/she had been the principal
investigator for any other studies of the health effects of passive smoking.
These additional studies were included in the interview if they met our inclusion
criteria. An appointment for the full telephone interview was scheduled.
Shortly before the scheduled interview, investigators were faxed a confirmation
letter that included a list of materials to have on hand during the interview
(a list of publications, the grant proposal, and any progress reports, unpublished
data summaries, talks, or abstracts). All interviews were conducted by one
of the authors (A.L.M.) between January 1996 and April 1997.
To determine the extent of unpublished results, we asked investigators
to identify all of their published and unpublished results on the health effects
of passive smoking. We also asked investigators about their publications,
study design characteristics, and reason(s) for unpublished results based
on an instrument used by Dickersin et al.1,24
To test our hypothesis that the statistical significance of results,
study design characteristics, and funding sources are associated with time
to publication, we classified studies as "published" or "unpublished" and
collected information on the variables listed below during the interviews.
Interview data were supplemented with data from the granting agency or the
first publication from the study.
Studies were excluded from the analysis of time to publication if (1)
data analysis had not yet begun as the investigators were not able to answer
our questions about the statistical significance of the data or final sample
size; (2) the publications associated with the study were published before
the start of the grant, as these publications could not be attributed to the
funded project; or (3) the publications associated with the study contained
no statistical analyses, as statistical significance could not be categorized.
The remaining studies were classified as "published" if any of the results
had been published in a peer-reviewed, non–peer-reviewed, or in-press
publication. Studies were classified as "unpublished" if none of the results
had been published or if results had been published only as abstracts, since
investigators were unable to provide copies of their abstracts or the abstracts
did not contain sufficient information.
Time to Publication.— Time to publication was calculated by subtracting the year funding began
from the year of publication of the first original research article on the
health effects of passive smoking resulting from the grant. Unpublished studies
were censored at the year the interview was conducted. The funding start date
was used to measure time to publication since the completion date was difficult
Statistical Significance of Data.— The statistical significance of each study was classified as "statistically
significant" (results indicate a statistically significant association between
passive smoking and a harmful health effect); "statistically nonsignificant"
(results do not indicate a statistically significant association); or "mixed"
(multiple primary outcomes were measured and at least 1 was statistically
significant). "Statistically significant" was defined as a P ≤.05, an odds ratio not overlapping 1, or other statistics typically
characterized as significant. For unpublished studies, the categorization
of statistical significance was based on the investigator's classification
of his/her unpublished results. For published studies, the categorization
was based on the significance of the results presented in the first original
research publication as determined by 1 of the authors (A.L.M.) and a doctorate
student in epidemiology.
Study Subjects.— Classified as "animals" or "humans."
Study Design.— A study was classified as "experimental" if a test regimen initiated
by the investigator was administered to a treatment group, but not a control
group, or as "observational" if the study was a cohort, case-control, or descriptive
Study Size.— Classified as "500 or less" or "more than 500" based on the distribution
of sample sizes.
Health Outcome.— Classified as "respiratory effects," "lung cancer," "cardiovascular
effects," "pregnancy outcome and/or birth defects," and "other" (includes
studies with mixed outcomes).
Funding Source and Length.—The primary funding source was identified by the investigator and was
categorized as "government," "private," "tobacco industry," or "other" and
as "external" (investigator not affiliated with funding agency) or "internal."
Length of funding was classified as "5 years or less" or "more than 5 years"
based on the distribution of funded studies.
The χ2 statistic was used to compare the characteristics
of participants and refusals. Predictors of time to publication were analyzed
by proportional hazards models.25 When there
was evidence against the proportional hazards assumptions,26P values were obtained by the nonparametric Wilcoxon test.27 Multivariate models were built using stepwise addition
of variables until no further additions provided a statistically significant
improvement in prediction at P =.05.
We identified 105 investigators eligible for inclusion. Four investigators
could not be located, 21 were found to be ineligible after completing the
preinterview, and 2 were eliminated after being interviewed because their
studies had no results on the health effects of passive smoking. The response
rate among the remaining 78 investigators was 83% (65/78). Characteristics
of participants and refusals are shown in Table 1. Of the 13 investigators who chose not to participate, 5
cited a lack of time, 3 felt uncomfortable participating, and 5 did not state
The 65 participants provided information on 84 studies. Sixty-six (79%)
of these studies were identified based on the information from granting organizations
and 18 (21%) were identified by the investigators.
Fifty-nine (70%) of the 84 studies had some unpublished results. The
reasons stated most frequently for unpublished results were ongoing data collection
or analysis (n=33 times), lack of time (n=26), and competing priorities (n=11),
such as other recently funded studies, studies where passive smoking was a
minor component, and career changes. Statistically nonsignificant results
were cited as a reason for failure to publish for only 2 studies. One investigator
stated that they chose a less prestigious journal to publish their statistically
nonsignificant results. Four manuscripts (2 with statistically significant
and 2 with statistically nonsignificant results) resulting from 3 studies
were unpublished because they had been rejected from a journal. These manuscripts
were being resubmitted.
Of the 84 identified studies, 23 were excluded from the analysis of
time to publication because data analysis had not begun (n=19), the publications
were published before the start of the grant (n=2), or the publications had
no statistical analyses (n=2). Of the remaining 61 studies, 47 were classified
as "published" and 14 as "unpublished."
Figure 1 shows that the survival
curve plots for time to publication for all studies differed by the statistical
significance of the study results (P=.004). The median
time to publication was 5 years (95% confidence interval [CI], 4-7 years)
for statistically nonsignificant studies (n=21) and 3 years (95% CI, 3-5 years)
for statistically significant studies (n=33). Only 1 of the 7 studies with
mixed results was published; the time to publication was 6 years.
As shown in Table 2, univariate
analysis of all studies showed that statistically significant results, experimental
study design, study size of 500 or less, and animals as subjects were predictive
of time to publication. When the studies with human participants were analyzed
separately (Table 2), only the
statistical significance of the results was predictive of time to publication.
Multivariate analysis of all studies indicated that statistical significance
(P=.001) and study design (P=.01)
were the only independent predictors of time to publication (P>.16 for all other variables). Multivariate analysis of the human
studies showed that only statistical significance was predictive of time to
publication (P=.007) (P>.15
for all other variables).
Our results indicate that statistically significant studies are published
sooner than statistically nonsignificant studies, as reported for clinical
studies.19,28 This finding cannot
be attributed to differences in the sample sizes, funding sources, or health
outcomes measured. Our finding of a publication delay can be partially explained
by the reasons that investigators cited for not publishing.
The delay in the publication of statistically nonsignificant results
has implications for conducting reviews and meta-analyses. The time lag in
publication argues for regular updating of reviews, as practiced by the Cochrane
Collaboration,29 as statistical significance
of a review could change over time. Furthermore, defining eligibility for
inclusion in a review by the start date of a study could reduce bias in reviews
by including unpublished results. Since 47 (77%) of the 61 studies we identified
had at least some published results, researchers conducting reviews on passive
smoking should be aware of these studies and seek data from them.
One limitation of our study is that we interviewed only English-speaking
investigators funded by US organizations. Therefore, we may have underestimated
the number of studies of the health effects of passive smoking. A second limitation
is that we had to rely on principal investigators to tell us the statistical
significance of their unpublished results. Finally, our ability to identify
studies was limited by the inadequate record keeping and willingness to share
information on the part of funding agencies or investigators. For example,
only half of the tobacco-affiliated organizations responded to our multiple
requests for information. Our difficulty with identifying unpublished studies
supports the need to register prospectively observational studies, as has
been suggested for clinical trials.30
Misakian AL, Bero LA. Publication Bias and Research on Passive SmokingComparison of Published and Unpublished Studies. JAMA. 1998;280(3):250-253. doi:10.1001/jama.280.3.250