Author Affiliations: Division of Emergency Medicine, University of California, San Francisco (Drs Callaham and Weber), Department of Emergency Medicine, University of Florida at Jacksonville, Jacksonville (Dr Wears).
Context Citation by other authors is important in the dissemination of published
science, but factors predicting it are little studied.
Methods To identify characteristics of published research predicting citation
in other journals, we searched the Science Citations Index database for a
standardized 3.5 years for all citations of published articles originally
submitted to a 1991 emergency medicine specialty meeting. Analysis was conducted
by classification and regression trees, a nonparametric modeling technique
of regression trees, to determine the impact of previously determined characteristics
of the full articles on the outcome measures. We calculated the the number
of times an article was cited each year and calculated the mean impact factor
(citations per manuscript per year) in other citing journals.
Results Of the 493 submitted manuscripts, 204 published articles met entry criteria.
The mean citations per year was 2.04 (95% confidence interval, 1.6-2.4; range,
0-20.9) in 440 different journals. Nineteen articles (9.3%) were never cited.
The ability to predict the citations per year was weak (pseudo R2 = 0.14.). The strongest predictor of citations per year
was the impact factor of the original publishing journal. The presence of
a control group, the subjective newsworthiness score, and sample size predicted
citation frequency (24.3%, 26.0%, and 26.5% as strongly, respectively). The
ability to predict mean impact factor of the citing journals was even weaker
(pseudo R2 = 0.09). The impact factor
of the publishing journal was the strongest predictor, followed by the newsworthiness
score (89.9% as strongly) and a subjective quality score (61.5%). Positive
outcome bias was not evident for either outcome measure.
Conclusion In this cohort of published research, commonly used measures of study
methodology and design did not predict the frequency of citations or the importance
of citing journals. Positive outcome bias was not evident. The impact factor
of the original publishing journal was more important than any other variable,
suggesting that the journal in which a study is published may be as important
as traditional measures of study quality in ensuring dissemination.
Although publication is a crucial portion of the scientific process,
an equally important part is the subsequent use and citation of these published
articles by other researchers and authors. We studied a cohort of all research
submitted to a scientific meeting and subsequently published to determine
how these studies were cited by other authors and determine what characteristics
(including positive results) were associated with more frequent citation.
We previously reported the methods of the first phase of this study.1 To summarize, all abstracts of scientific studies
submitted to the Society for Academic Emergency Medicine (SAEM) meeting in
1991 were examined. Each submitted abstract was categorized in a blinded fashion
according to research design, number of subjects, and other characteristics
(Table 1 and Table 2), and rated subjectively for scientific quality and newsworthiness
using a modified delphi method. Searches of MEDLINE, EMBASE, and Cochrane
databases were conducted for 4 years after the meeting to identify publication
in any journal listed in the National Library of Medicine (NLM). If needed,
further information was obtained from authors in writing.2
The characteristics of submitted abstracts that predicted full publication
were reported previously.1
In March 2000, the cohort was further examined by searching the Science
Citation Index (SCI) database for articles of every study submitted to this
meeting that had been published in full (http://www.webofscience.com). For each published article, all citations of that article from publication
to the time of the search were identified. Citations were minimal until 2
years after publication, so results were analyzed beginning then and for the
next 3.5 years.
Two outcome measures were determined. The number of citations per year
during the study period was calculated for each article. This can be considered
the article's "impact factor," analogous to the traditional impact factor
for journals (the annual number of citations per published article in a journal).3 The second outcome measure was the mean citing journal
impact factor (CJIF) of each published article, calculated by averaging the
impact factors of each journal citing that publicaton. Together, these 2 outcomes
provide estimates of both the quantity and quality of citations of a manuscript.
Summary statistics were calculated using Systat 9.02 statistical software
(SPSS, Evanston, Ill). The relationship of the primary outcome measures (citations
per year and CJIF) to characteristics of the original abstracts was assessed
using regression trees (CART, or Classification and Regression Trees).4 Exploratory analyses of the relationships of interest
in this study suggested that simpler multiple regression models would not
fit the data as well as regression trees, which are particularly suited to
complex interactions among predictors.
The analysis was conducted using CART 4.0 (Salford Systems, San Diego,
Calif). The candidate predictor variables were impact factor of the publishing
journal, number of subjects (in quartiles, 0-33, 34-101, 102-425, and >425),
subjects (human, animal, or other), binary variables representing presence
or absence of an explicit hypothesis, control group, blinding, acceptance
for presentation at the research meeting, method (prospective or retrospective),
and positive results (as previously defined1),
and the subjective "newsworthiness" and quality scores, derived from the full
article, not the abstract.
The model was allowed to continue splitting until a node contained 20
cases or fewer (about 10% of the sample) and the resultant trees were tested
by 10-fold cross-validation analysis to eliminate over-fit trees and to identify
the best model. The cross-validation relative error was used as a pseudo-R2 measure (analogous to but not truly R2), a global measure of the final model's explanatory
power on a new data set (a more detailed description of the methodology is
available from the authors).
This study was approved by the committee on human research of the University
Four hundred ninety-three studies were submitted to the 1991 SAEM meeting
from a total of 144 institutions and 103 medical schools, of which 179 (36%)
were accepted for presentation at the meeting. Two hundred nineteen (44%)
of the 493 studies submitted to this meeting were published in 44 peer-reviewed
journals (37 in specialties other than emergency medicine) with impact factors
ranging from 0.23 to 24.5.1
Fifteen of the published articles appeared in journals that are not
SCI citation sources and thus have no SCI impact factor. These were excluded
from subsequent analysis, leaving 204 published articles.
Of the 204 included published articles, 19 (9.3%) had no citations during
the study period; the remainder were cited a total of 1446 times by 440 different
journals with impact factors ranging from 0.01 to 24.5, 434 of them from disciplines
other than emergency medicine. Seventy-nine studies (39%) were cited 1 or
2 times only during the study period. The mean citations per year was 2.04
(95% confidence interval [CI], 1.6-2.4; range, 0-20.9) and the mean impact
factor of the citing journals was 1.69 (95% CI, 1.50-1.87; range, 0.01-9.9).
Citing journals included all the large general medical journals; 18% of citing
journals had an impact factor greater than 3, compared with only 10% of all
journals in the SCI.5
Univariate analysis is not reported for purposes of brevity. CART regression
for citations per year yielded a pseudo-R2
statistic of 0.14, suggesting relatively low explanatory power. The regression
tree suggested that impact factor was the only variable of importance (Table 1); other characteristics of the
studies either had no influence, or were almost completely subsumed by the
impact factor of the publishing journal. After adjustment for the impact factor
of the publishing journal, presence of a control group, the subjective "newsworthiness"
score, and sample size were the next most important determinants of citation
The regression analysis for CJIF yielded a pseudo-R2 statistic of 0.09, also suggesting little predictive
ability. The impact factor of the publishing journal was the most important
variable here also; however, in contrast to citations per year, the newsworthiness
and quality scores also contributed substantially (Table 2).
Positive outcome bias was not evident in this sample in either univariate
analysis or the regression model. Full manuscripts with negative outcomes
had a mean 1.96 citations per year (95% CI, 1.2-2.7), whereas those with positive
outcomes had 1.84 citations per year (95% CI, 1.2-2.5). Results for CJIF were
The publication of research in peer reviewed journals is only an intermediate
outcome, satisfying to authors but not necessarily useful to others. There
is no way to measure how useful a published article is to clinicians, but
we can measure its impact on other authors by how frequently they cite it
in their publications. Such citations are a hallmark of academic achievement
for authors and for journals, correlate highly with the opinion of peers as
to a scientist's contribution to his/her field and are used by medical school
deans for promotion reviews.6,7
Citations complete the chain of publication and that underpins the evolution
of scientific knowledge.
Of the 204 publications studied, 185 were cited a total of 1446 times
by 440 different journals. The average article was cited 2.04 times per year,
and the mean citing journal impact factor was 1.84. This is roughly comparable
to the citation rate of all material published in all journals in the SCI,8 and the proportion of citing journals with high impact
factors in our sample was greater than that of the SCI as a whole.5
Only 19 (9.3%) of these studies were never cited during the study period
(even by their own authors), compared with 22% for all the international medical
literature,9 11% of AIDS articles, and 15%
of biology articles.10 These figures suggest
that the study cohort was broadly representative of the biomedical literature.
We found that the impact factor of the original publishing journal,
not the methodology or quality of the research, was the strongest predictor
of citations per year. A pessimist would suggest that despite the era of accessibility
due to electronic searching and retrieval, citation may be more strongly influenced
by the reputation of the publishing journal than by the design merits of the
study. Thus, a strong or seminal paper submitted to a minor journal might
not receive the scientific recognition it deserves. Likewise, a weak article
published in a major journal will probably receive more recognition than it
An optimist might interpret our results to mean that journals are perfectly
efficient in publishing studies of a uniform quality, with the same citation
value as the journal itself. However, the SCI reports marked variability in
citation rates of individual articles in any given journal,5
and we found no relationship between study design (and other measures of quality),
and the impact factor of the original publishing journal.1
Once the impact factor of the publishing journal was accounted for,
the subjective newsworthiness score (from a delphi panel rating), sample size
and presence of a control group were the only major predictors of citations
(and whose respective contribution was only 26.0%, 26.5%, and 24.3% as strongly).
This is disappointing since one would hope that characteristics of sound design
would be the most important predictors. However, it may also indicate that
quality checklists for assessing studies are not meaningful or accurate11,12 or that editors are able to identify
importance and originality in research independent of the "plumbing" of study
It is encouraging that studies with positive results were not cited
more frequently or by more prestigious journals. This is in contrast to the
consistent bias toward acceptance and publication of studies with positive
results previously reported for this study cohort and others.1,13- 20
Although documented at earlier steps of the publication process (from submission
to research meetings to publication of full articles), the final step, citation
by subsequent authors, appears to be relatively free of this bias.
In this study, acceptance of research for presentation at the meeting
had no predictive power for citation frequency or impact factor of citing
journals. As with our previous finding that acceptance failed to predict publication
of a manuscript or the impact factor of the publishing journal,1
the current results suggest methods of screening research for meeting presentation
are very fallible.
Our study has several limitations. It examined research (mostly clinical)
from only 1 meeting although this cohort has previously been shown to share
key characteristics with research from other specialty meetings1
and was published in 44 journals covering a broad spectrum of specialties
and impact factors. Eighty-four percent of publishing journals and 98% of
the citing journals were from specialties other than that sponsoring the meeting.
Citations per year is not an ideal tool for detecting positive outcome bias
because a citation may not credit or praise a study but instead refute or
criticize it. However, most citations in most articles do not refute the cited
paper, and authors, journals, and promotion committees consider the number
of citations to constitute a measure of merit. Finally, the ability of the
regression model to predict outcomes was poor, meaning that most of the variance
was accounted for by unmeasured variables (such as, hopefully, the relevance
of the study to the citing research).
Callaham M, Wears RL, Weber E. Journal Prestige, Publication Bias, and Other Characteristics Associated With Citation of Published Studies in Peer-Reviewed Journals. JAMA. 2002;287(21):2847–2850. doi:10.1001/jama.287.21.2847