Citation Inaccuracies in Influential Surgical Journals | Medical Journals and Publishing | JAMA Surgery | JAMA Network
[Skip to Navigation]
Sign In
Table.  Characteristics of Citing and Referenced Articles
Characteristics of Citing and Referenced Articles
1.
Wager  E, Middleton  P.  Technical editing of research reports in biomedical journals.   Cochrane Database Syst Rev. 2008;(4):MR000002. doi:10.1002/14651858.MR000002.pub3Google Scholar
2.
Jergas  H, Baethge  C.  Quotation accuracy in medical journal articles-a systematic review and meta-analysis.   PeerJ. 2015;3:e1364. doi:10.7717/peerj.1364PubMedGoogle Scholar
3.
Harzing  A-W.  Are our referencing errors undermining our scholarship and credibility? the case of expatriate failure rates.   J Organ Behav. 2002;23(1):127-148. doi:10.1002/job.125Google ScholarCrossref
4.
Merton  RK.  The Matthew effect in science: the reward and communication systems of science are considered.   Science. 1968;159(3810):56-63. doi:10.1126/science.159.3810.56PubMedGoogle ScholarCrossref
5.
Simkin  MV, Roychowdhury  VP.  Read before you cite!   arXiv. Posted online December 3, 2002. https://arxiv.org/abs/cond-mat/0212043v1Google Scholar
6.
Larsson  KS.  The dissemination of false data through inadequate citation.   J Intern Med. 1995;238(5):445-450. doi:10.1111/j.1365-2796.1995.tb01222.xPubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Research Letter
    May 26, 2021

    Citation Inaccuracies in Influential Surgical Journals

    Author Affiliations
    • 1Surgeons Writing About Trauma, Stanford University, Stanford, California
    • 2School of Medicine, Stanford University, Stanford, California
    • 3Department of Epidemiology and Population Health, Stanford University, Stanford, California
    • 4Department of Surgery, Stanford University, Stanford, California
    JAMA Surg. 2021;156(8):791-792. doi:10.1001/jamasurg.2021.1445

    Meta-analyses report 20% to 25% citation inaccuracy rates in biomedical literature.1,2 Citation inaccuracies risk amplifying misinformation and undermining research validity.3 While many inaccuracies are likely blunders, patterns such as highly cited articles garnering repeated citations via the Matthew effect deserve recognition.4

    We aimed to investigate the prevalence and potential etiologies of citation inaccuracies in surgical literature by systematically characterizing influential surgical articles and references they cited. We hypothesized that citation inaccuracies in influential surgical journals are common but hoped that a simple checklist may facilitate accurate surgical science dissemination.

    Methods

    A random number generator selected 20 of 100 most-cited articles from each of Annals of Surgery, British Journal of Surgery, and JAMA Surgery from January 2003 to December 2007 and January 2013 to December 2017. Three reviewers (J.C., A.G., and A.K.) independently evaluated full texts and performed consensus characterization of 120 citing articles and 10 of their randomly selected references (1200 referenced articles) (eMethods in the Supplement). This study was exempt from institutional review board review because no human participants were involved.

    We defined 4 inaccuracy types: (1) secondary citation: duplicate of another cited fact in referenced article; (2) fact not found: statements, direct or implied, not supported by referenced article; (3) inaccurate statistics/population: statistical misrepresentations or extrapolating findings to vastly different populations; and (4) contradictory conclusion: claims contradicting referenced article conclusion. Cohen h (binary characteristics), Fisher exact test, and the t test evaluated associations between articles with citation inaccuracies and article or author characteristics. Two-sided P values were statistically significant at .05.

    Results
    Article Characteristics

    The 120 citing articles were cited a mean (SD) of 216 (181) times (Table). Among 120 first authors, 10 (8%) were trainees and 76 (63%) were affiliated with institutions in majority English–speaking countries. Citing articles had a mean (SD) of 51 (36) references and 4 (5) self-citations. Referenced articles had a mean (SD) of 156 (96) citations when they were referenced. Nearly 70% (827 of 1200) of referenced facts were found in abstracts.

    Error Characteristics

    We found overall 9% (111 of 1200) prevalence of citation inaccuracies. Forty-three percent (51 of 120) of articles had no citation inaccuracies; 32% (38 of 120) had 1, 20% (24 of 120) had 2, and 6% (7 of 120) had 3 or 4 inaccuracies. The most common inaccuracy was facts not found, followed by inaccurate statistics/populations, secondary citation copying, and contradictory conclusions. References validating findings (10 of 56 [18%]) were most commonly erroneous, followed by those substantiating scope of problem (44 of 342 [13%]), other references (52 of 637 [8%]), and procedure or methods (5 of 108 [5%]).

    Associations With Citation Inaccuracies

    Articles with and without citation inaccuracies did not have statistically significant differences in first author (trainee status [h = 0.03; P = .99], advanced degree [h = 0.36; P = .08], affiliation with institution in majority English–speaking country [h = 0.16; P = .49]) or article characteristics (publication period [h = 0.03; P = .99], subspecialty [P = .32]), type [P = .46], design [P = .47], number of citations [P = .50]) (eTable in the Supplement). We did not find statistically significant associations at any inaccuracy count cutoff or at the referenced article level.

    Discussion

    We found a 9% citation inaccuracy rate among highly cited articles in influential surgical journals. Half of inaccuracies constituted facts not supported by, and a few directly contradicting, referenced articles’ findings. Evaluated articles represent work that underwent rigorous peer review; citation inaccuracies in surgical literature are likely more widespread.

    Accurate citations provide critical framework for disseminating knowledge, yet citation accuracy is likely assumed too often. Article submission instructions do not prompt authors to fact-check citations, many peer reviewers and principal investigators likely assume citations were fact-checked, and few authors may actually read articles they cite.5 Highly cited articles may suggest credibility, but allowing citation tally in and of itself to lend credibility may disseminate unchecked misinterpretations. Anyone can make citation errors; a simple checklist may help authors and peer reviewers verify citation accuracy (Box). Simple practices such as reading articles before referencing would have prevented errors we identified.

    Box Section Ref ID
    Box.

    Checklist to Mitigate Citation Inaccuracies

    Consider
    • Using a reference manager (eg, Zotero, Endnote)

    • Citing material after reading the body of the article

    • Citing primary sourcea

    • Confirming referenced article’s population can be extrapolated; consider including enough information to allow readers to judge for themselves

    • Being explicit when citing animal studies

    • Double-checking important citations (principal investigators and peer reviewers)

    Avoid
    • Manually updating references

    • Citing based on title or abstract alone

    • Citing secondary citation (ie, reference of a reference) without reading primary source

    • Substantiating findings or scope of problem from different populations

    • Suggesting animal studies reflect clinical practice

    • Assuming citations are accurate

    a Citing evidence synthesis after reading would be appropriate.

    Not all citation inaccuracies are detrimental to science. However, if a substantial claim is erroneously cited, the claim is repeatedly cited without fact-checking, and sufficient enough citations accumulate to establish unchecked credibility, a misinterpretation may lead to consequential affirmation of misinformation.6 Citations may be rearmost in text, but citation accuracy should not be an afterthought.

    Back to top
    Article Information

    Corresponding Author: Jeff Choi, MD, MSc, Stanford University, 300 Pasteur Dr, H3591, Stanford, CA 94305 (jc2226@stanford.edu).

    Accepted for Publication: March 4, 2021.

    Published Online: May 26, 2021. doi:10.1001/jamasurg.2021.1445

    Author Contributions: Dr Choi had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Choi, Gupta, Kaghazchi, Spain.

    Acquisition, analysis, or interpretation of data: Choi, Gupta, Kaghazchi, Htwe, Baiocchi.

    Drafting of the manuscript: Choi, Gupta, Kaghazchi, Htwe.

    Critical revision of the manuscript for important intellectual content: Kaghazchi, Baiocchi, Spain.

    Statistical analysis: Choi, Gupta, Baiocchi.

    Administrative, technical, or material support: Htwe, Spain.

    Supervision: Spain.

    Conflict of Interest Disclosures: None reported.

    Additional Contributions: We would like to thank the Neil and Claudia Doerhoff Fund for support of Dr Choi’s scholarly activities. The Neil and Claudia Doerhoff Fund has no role in Dr Choi’s activities besides funding.

    References
    1.
    Wager  E, Middleton  P.  Technical editing of research reports in biomedical journals.   Cochrane Database Syst Rev. 2008;(4):MR000002. doi:10.1002/14651858.MR000002.pub3Google Scholar
    2.
    Jergas  H, Baethge  C.  Quotation accuracy in medical journal articles-a systematic review and meta-analysis.   PeerJ. 2015;3:e1364. doi:10.7717/peerj.1364PubMedGoogle Scholar
    3.
    Harzing  A-W.  Are our referencing errors undermining our scholarship and credibility? the case of expatriate failure rates.   J Organ Behav. 2002;23(1):127-148. doi:10.1002/job.125Google ScholarCrossref
    4.
    Merton  RK.  The Matthew effect in science: the reward and communication systems of science are considered.   Science. 1968;159(3810):56-63. doi:10.1126/science.159.3810.56PubMedGoogle ScholarCrossref
    5.
    Simkin  MV, Roychowdhury  VP.  Read before you cite!   arXiv. Posted online December 3, 2002. https://arxiv.org/abs/cond-mat/0212043v1Google Scholar
    6.
    Larsson  KS.  The dissemination of false data through inadequate citation.   J Intern Med. 1995;238(5):445-450. doi:10.1111/j.1365-2796.1995.tb01222.xPubMedGoogle ScholarCrossref
    ×