eTable. Association between articles with and without citation errors and first author/article characteristics
Customize your JAMA Network experience by selecting one or more topics from the list below.
Identify all potential conflicts of interest that might be relevant to your comment.
Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.
Err on the side of full disclosure.
If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.
Not all submitted comments are published. Please see our commenting policy for details.
Choi J, Gupta A, Kaghazchi A, Htwe TS, Baiocchi M, Spain DA. Citation Inaccuracies in Influential Surgical Journals. JAMA Surg. 2021;156(8):791–792. doi:10.1001/jamasurg.2021.1445
Meta-analyses report 20% to 25% citation inaccuracy rates in biomedical literature.1,2 Citation inaccuracies risk amplifying misinformation and undermining research validity.3 While many inaccuracies are likely blunders, patterns such as highly cited articles garnering repeated citations via the Matthew effect deserve recognition.4
We aimed to investigate the prevalence and potential etiologies of citation inaccuracies in surgical literature by systematically characterizing influential surgical articles and references they cited. We hypothesized that citation inaccuracies in influential surgical journals are common but hoped that a simple checklist may facilitate accurate surgical science dissemination.
A random number generator selected 20 of 100 most-cited articles from each of Annals of Surgery, British Journal of Surgery, and JAMA Surgery from January 2003 to December 2007 and January 2013 to December 2017. Three reviewers (J.C., A.G., and A.K.) independently evaluated full texts and performed consensus characterization of 120 citing articles and 10 of their randomly selected references (1200 referenced articles) (eMethods in the Supplement). This study was exempt from institutional review board review because no human participants were involved.
We defined 4 inaccuracy types: (1) secondary citation: duplicate of another cited fact in referenced article; (2) fact not found: statements, direct or implied, not supported by referenced article; (3) inaccurate statistics/population: statistical misrepresentations or extrapolating findings to vastly different populations; and (4) contradictory conclusion: claims contradicting referenced article conclusion. Cohen h (binary characteristics), Fisher exact test, and the t test evaluated associations between articles with citation inaccuracies and article or author characteristics. Two-sided P values were statistically significant at .05.
The 120 citing articles were cited a mean (SD) of 216 (181) times (Table). Among 120 first authors, 10 (8%) were trainees and 76 (63%) were affiliated with institutions in majority English–speaking countries. Citing articles had a mean (SD) of 51 (36) references and 4 (5) self-citations. Referenced articles had a mean (SD) of 156 (96) citations when they were referenced. Nearly 70% (827 of 1200) of referenced facts were found in abstracts.
We found overall 9% (111 of 1200) prevalence of citation inaccuracies. Forty-three percent (51 of 120) of articles had no citation inaccuracies; 32% (38 of 120) had 1, 20% (24 of 120) had 2, and 6% (7 of 120) had 3 or 4 inaccuracies. The most common inaccuracy was facts not found, followed by inaccurate statistics/populations, secondary citation copying, and contradictory conclusions. References validating findings (10 of 56 [18%]) were most commonly erroneous, followed by those substantiating scope of problem (44 of 342 [13%]), other references (52 of 637 [8%]), and procedure or methods (5 of 108 [5%]).
Articles with and without citation inaccuracies did not have statistically significant differences in first author (trainee status [h = 0.03; P = .99], advanced degree [h = 0.36; P = .08], affiliation with institution in majority English–speaking country [h = 0.16; P = .49]) or article characteristics (publication period [h = 0.03; P = .99], subspecialty [P = .32]), type [P = .46], design [P = .47], number of citations [P = .50]) (eTable in the Supplement). We did not find statistically significant associations at any inaccuracy count cutoff or at the referenced article level.
We found a 9% citation inaccuracy rate among highly cited articles in influential surgical journals. Half of inaccuracies constituted facts not supported by, and a few directly contradicting, referenced articles’ findings. Evaluated articles represent work that underwent rigorous peer review; citation inaccuracies in surgical literature are likely more widespread.
Accurate citations provide critical framework for disseminating knowledge, yet citation accuracy is likely assumed too often. Article submission instructions do not prompt authors to fact-check citations, many peer reviewers and principal investigators likely assume citations were fact-checked, and few authors may actually read articles they cite.5 Highly cited articles may suggest credibility, but allowing citation tally in and of itself to lend credibility may disseminate unchecked misinterpretations. Anyone can make citation errors; a simple checklist may help authors and peer reviewers verify citation accuracy (Box). Simple practices such as reading articles before referencing would have prevented errors we identified.
Using a reference manager (eg, Zotero, Endnote)
Citing material after reading the body of the article
Citing primary sourcea
Confirming referenced article’s population can be extrapolated; consider including enough information to allow readers to judge for themselves
Being explicit when citing animal studies
Double-checking important citations (principal investigators and peer reviewers)
Manually updating references
Citing based on title or abstract alone
Citing secondary citation (ie, reference of a reference) without reading primary source
Substantiating findings or scope of problem from different populations
Suggesting animal studies reflect clinical practice
Assuming citations are accurate
a Citing evidence synthesis after reading would be appropriate.
Not all citation inaccuracies are detrimental to science. However, if a substantial claim is erroneously cited, the claim is repeatedly cited without fact-checking, and sufficient enough citations accumulate to establish unchecked credibility, a misinterpretation may lead to consequential affirmation of misinformation.6 Citations may be rearmost in text, but citation accuracy should not be an afterthought.
Corresponding Author: Jeff Choi, MD, MSc, Stanford University, 300 Pasteur Dr, H3591, Stanford, CA 94305 (firstname.lastname@example.org).
Accepted for Publication: March 4, 2021.
Published Online: May 26, 2021. doi:10.1001/jamasurg.2021.1445
Author Contributions: Dr Choi had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Choi, Gupta, Kaghazchi, Spain.
Acquisition, analysis, or interpretation of data: Choi, Gupta, Kaghazchi, Htwe, Baiocchi.
Drafting of the manuscript: Choi, Gupta, Kaghazchi, Htwe.
Critical revision of the manuscript for important intellectual content: Kaghazchi, Baiocchi, Spain.
Statistical analysis: Choi, Gupta, Baiocchi.
Administrative, technical, or material support: Htwe, Spain.
Conflict of Interest Disclosures: None reported.
Additional Contributions: We would like to thank the Neil and Claudia Doerhoff Fund for support of Dr Choi’s scholarly activities. The Neil and Claudia Doerhoff Fund has no role in Dr Choi’s activities besides funding.