[Skip to Content]
[Skip to Content Landing]
Figure 1.
Annual National Institutes of Health–Funded Publications by Country of Authors’ Affiliations, 2009 to 2017
Annual National Institutes of Health–Funded Publications by Country of Authors’ Affiliations, 2009 to 2017
Figure 2.
Annual National Institutes of Health–Funded Publications for 2009 to 2017
Annual National Institutes of Health–Funded Publications for 2009 to 2017

A, Top 10 non–US author affiliations. B, Top 10 low- and middle-income country author affiliations (excluding China).

Figure 3.
National Institutes of Health–Funded Publications and h Indices for 2017
National Institutes of Health–Funded Publications and h Indices for 2017

A, Top 20 countries of authors’ affiliations. B, Top 20 low- and middle-income countries of authors’ affiliations, excluding China (6982 affiliations; h index, 50).

Table.  
Top 10 Most Cited National Institutes of Health–Funded Publications With Non-US Coauthors, 2009-2017
Top 10 Most Cited National Institutes of Health–Funded Publications With Non-US Coauthors, 2009-2017
1.
National Institutes of Health. Impact of NIH research: our society. https://www.nih.gov/about-nih/what-we-do/impact-nih-research/our-society. Updated May 1, 2018. Accessed November 20, 2018.
2.
Viergever  RF, Hendriks  TCC.  The 10 largest public and philanthropic funders of health research in the world: what they fund and how they distribute their funds.  Health Res Policy Syst. 2016;14:12. doi:10.1186/s12961-015-0074-zPubMedGoogle ScholarCrossref
3.
National Institutes of Health Office of Budget. National Institutes of Health operating plan for FY 2019. https://officeofbudget.od.nih.gov/pdfs/FY20/cy/FY-2019-NIH-Operating-Plan-2.19.19.pdf. Published February 2018. Accessed November 28, 2018.
4.
Heggeness  ML, Evans  L, Pohlhaus  JR, Mills  SL.  Measuring diversity of the National Institutes of Health–funded workforce.  Acad Med. 2016;91(8):1164-1172. doi:10.1097/ACM.0000000000001209PubMedGoogle ScholarCrossref
5.
He  ZL, Geng  XS, Campbell-Hunt  C.  Research collaboration and research output: a longitudinal study of 65 biomedical scientists in a New Zealand university.  Res Policy. 2009;38(2):306-317. doi:10.1016/j.respol.2008.11.011Google ScholarCrossref
6.
Gazni  A, Didegah  F.  Investigating different types of research collaboration and citation impact: a case study of Harvard University’s publications.  Scientometrics. 2011;87(2):251-265. doi:10.1007/s11192-011-0343-8Google ScholarCrossref
7.
Wagner  CS, Staheli  L, Silberglitt  R, Wong  A, Kadtke  J. Linking effectively: learning lessons from successful collaboration in science and technology. RAND Corporation. https://www.rand.org/content/dam/rand/pubs/documented_briefings/2005/DB345.pdf. Published April 2002. Accessed November 28, 2018.
8.
Tariot  PN, Lopera  F, Langbaum  JB,  et al; Alzheimer’s Prevention Initiative.  The Alzheimer’s Prevention Initiative Autosomal-Dominant Alzheimer’s Disease Trial: a study of crenezumab versus placebo in preclinical PSEN1 E280A mutation carriers to evaluate efficacy and safety in the treatment of autosomal-dominant Alzheimer’s disease, including a placebo-treated noncarrier cohort.  Alzheimers Dement (N Y). 2018;4:150-160. doi:10.1016/j.trci.2018.02.002PubMedGoogle Scholar
9.
Fogarty International Center. Fogarty trainees study Colombian family for clues to prevent Alzheimer’s. Global Health Matters Newsletter. May/June 2017;16(3). https://www.fic.nih.gov/News/Publications/Pages/fogarty-trainees-study-colombian-family-prevent-alzheimers.aspx. Accessed November 28, 2018.
10.
Khan  K, Ahmed  E, Factor-Litvak  P,  et al.  Evaluation of an elementary school-based educational intervention for reducing arsenic exposure in Bangladesh.  Environ Health Perspect. 2015;123(12):1331-1336. doi:10.1289/ehp.1409462PubMedGoogle ScholarCrossref
11.
Zheng  Y, Walsh  T, Gulsuner  S,  et al.  Inherited breast cancer in Nigerian women.  J Clin Oncol. 2018;36(28):2820-2825. doi:10.1200/JCO.2018.78.3977PubMedGoogle ScholarCrossref
12.
Adams  JD; National Bureau of Economic Research.  Scientific Teams and Institutional Collaborations Evidence From US Universities, 1981-1999. Cambridge, MA: National Bureau of Economic Research; 2004. NBER working paper series working paper 10640. https://papers.nber.org/papers/W10640. Accessed November 28, 2018.
13.
Frenken  K, Hoekman  J, Kok  S, Ponds  R, van Oort  F, van Vliet  J. Death of distance in science? a gravity approach to research collaboration. In: Pyka  A, Scharnhorst  A, eds.  Innovation Networks: New Approaches in Modelling and Analyzing. Berlin, Heidelberg, Germany: Springer Berlin Heidelberg; 2009:43-57. doi:10.1007/978-3-540-92267-4_3
14.
Rigby  J.  Systematic grant and funding body acknowledgement data for publications: new dimensions and new controversies for research policy and evaluation.  Res Eval. 2011;20(5):365-375. doi:10.3152/095820211X13164389670392Google ScholarCrossref
15.
Abramo  G, D’Angelo  CA, Di Costa  F.  National research assessment exercises: a comparison of peer review and bibliometrics rankings.  Scientometrics. 2011;89(3):929-941. doi:10.1007/s11192-011-0459-xGoogle ScholarCrossref
16.
Abramo  G, D’Angelo  CA.  Evaluating research: from informed peer review to bibliometrics.  Scientometrics. 2011;87(3):499-514. doi:10.1007/s11192-011-0352-7Google ScholarCrossref
17.
National Science Board. Science & engineering indicators 2018. National Science Foundation. https://www.nsf.gov/statistics/state-indicators. Published January 15, 2018. Accessed November 27, 2018.
18.
Clarivate Analytics. InCites indicators handbook. http://help.incites.clarivate.com/inCites2Live/8980-TRS/version/default/part/AttachmentData/data/InCites-Indicators-Handbook%20-%20June%202018.pdf. Published June 2018. Accessed November 27, 2018.
19.
National Institutes of Health Office of Budget. History of Congressional appropriations, fiscal years 2000-2018. https://officeofbudget.od.nih.gov/pdfs/FY19/Approp%20History%20by%20IC%20FY%202000%20-%20FY%202018.pdf. Published February 2019. Accessed April 22, 2019.
20.
Web of Science. Web of Science core collection: funding information. https://clarivate.libguides.com/woscc/funding. Updated September 12, 2019. Accessed October 19, 2019.
21.
World Bank. World Bank country and lending groups. https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups. Published June 2018. Accessed December 3, 2018.
22.
Hirsch  JE.  An index to quantify an individual’s scientific research output.  Proc Natl Acad Sci U S A. 2005;102(46):16569-16572. doi:10.1073/pnas.0507655102PubMedGoogle ScholarCrossref
23.
Jacsó  P.  The h-index for countries in Web of Science and Scopus.  Online Inf Rev. 2009;33(4):831-837. doi:10.1108/14684520910985756Google ScholarCrossref
24.
Csajbók  E, Berhidi  A, Vasas  L, Schubert  A.  Hirsch-index for countries based on essential science indicators data.  Scientometrics. 2007;73(1):91-117. doi:10.1007/s11192-007-1859-9Google ScholarCrossref
25.
Adams  J.  Early citation counts correlate with accumulated impact.  Scientometrics. 2005;63(3):567-581. doi:10.1007/s11192-005-0228-9Google ScholarCrossref
26.
Abramo  G, D’Angelo  CA, Cicero  T.  What is the appropriate length of the publication period over which to assess research performance?  Scientometrics. 2012;93(3):1005-1017. doi:10.1007/s11192-012-0714-9Google ScholarCrossref
27.
Hanahan  D, Weinberg  RA.  Hallmarks of cancer: the next generation.  Cell. 2011;144(5):646-674. doi:10.1016/j.cell.2011.02.013PubMedGoogle ScholarCrossref
28.
Tamura  K, Peterson  D, Peterson  N, Stecher  G, Nei  M, Kumar  S.  MEGA5: Molecular Evolutionary Genetics Analysis using maximum likelihood, evolutionary distance, and maximum parsimony methods.  Mol Biol Evol. 2011;28(10):2731-2739. doi:10.1093/molbev/msr121PubMedGoogle ScholarCrossref
29.
Tamura  K, Stecher  G, Peterson  D, Filipski  A, Kumar  S.  MEGA6: Molecular Evolutionary Genetics Analysis version 6.0.  Mol Biol Evol. 2013;30(12):2725-2729. doi:10.1093/molbev/mst197PubMedGoogle ScholarCrossref
30.
Adams  PD, Afonine  PV, Bunkóczi  G,  et al.  PHENIX: a comprehensive Python-based system for macromolecular structure solution.  Acta Crystallogr D Biol Crystallogr. 2010;66(pt 2):213-221. doi:10.1107/S0907444909052925PubMedGoogle ScholarCrossref
31.
Li  H, Handsaker  B, Wysoker  A,  et al; 1000 Genome Project Data Processing Subgroup.  The Sequence Alignment/Map format and SAMtools.  Bioinformatics. 2009;25(16):2078-2079. doi:10.1093/bioinformatics/btp352PubMedGoogle ScholarCrossref
32.
Schindelin  J, Arganda-Carreras  I, Frise  E,  et al.  Fiji: an open-source platform for biological-image analysis.  Nat Methods. 2012;9(7):676-682. doi:10.1038/nmeth.2019PubMedGoogle ScholarCrossref
33.
Ronquist  F, Teslenko  M, van der Mark  P,  et al.  MrBayes 3.2: efficient Bayesian phylogenetic inference and model choice across a large model space.  Syst Biol. 2012;61(3):539-542. doi:10.1093/sysbio/sys029PubMedGoogle ScholarCrossref
34.
ENCODE Project Consortium.  An integrated encyclopedia of DNA elements in the human genome.  Nature. 2012;489(7414):57-74. doi:10.1038/nature11247PubMedGoogle ScholarCrossref
35.
Kumar  S, Stecher  G, Tamura  K.  MEGA7: Molecular Evolutionary Genetics Analysis version 7.0 for bigger data sets.  Mol Biol Evol. 2016;33(7):1870-1874. doi:10.1093/molbev/msw054PubMedGoogle ScholarCrossref
36.
Grabherr  MG, Haas  BJ, Yassour  M,  et al.  Full-length transcriptome assembly from RNA-Seq data without a reference genome.  Nat Biotechnol. 2011;29(7):644-652. doi:10.1038/nbt.1883PubMedGoogle ScholarCrossref
37.
Auton  A, Brooks  LD, Durbin  RM,  et al; 1000 Genomes Project Consortium.  A global reference for human genetic variation.  Nature. 2015;526(7571):68-74. doi:10.1038/nature15393PubMedGoogle ScholarCrossref
38.
Uhlén  M, Fagerberg  L, Hallström  BM,  et al.  Tissue-based map of the human proteome.  Science. 2015;347(6220):1260419. doi:10.1126/science.1260419PubMedGoogle Scholar
39.
Vos  T, Barber  RM, Bell  B,  et al; Global Burden of Disease Study 2013 Collaborators.  Global, regional, and national incidence, prevalence, and years lived with disability for 301 acute and chronic diseases and injuries in 188 countries, 1990-2013: a systematic analysis for the Global Burden of Disease Study 2013.  Lancet. 2015;386(9995):743-800. doi:10.1016/S0140-6736(15)60692-4PubMedGoogle ScholarCrossref
40.
Kundaje  A, Meuleman  W, Ernst  J,  et al; Roadmap Epigenomics Consortium.  Integrative analysis of 111 reference human epigenomes.  Nature. 2015;518(7539):317-330. doi:10.1038/nature14248PubMedGoogle ScholarCrossref
41.
GTEx Consortium.  The Genotype-Tissue Expression (GTEx) pilot analysis: multitissue gene regulation in humans.  Science. 2015;348(6235):648-660. doi:10.1126/science.1262110PubMedGoogle ScholarCrossref
42.
Fitzmaurice  C, Dicker  D, Pain  A,  et al; Global Burden of Disease Cancer Collaboration.  The global burden of cancer 2013.  JAMA Oncol. 2015;1(4):505-527. doi:10.1001/jamaoncol.2015.0735PubMedGoogle ScholarCrossref
43.
Lundgren  JD, Babiker  AG, Gordin  F,  et al; INSIGHT START Study Group.  Initiation of antiretroviral therapy in early asymptomatic HIV infection.  N Engl J Med. 2015;373(9):795-807. doi:10.1056/NEJMoa1506816PubMedGoogle ScholarCrossref
44.
Heneka  MT, Carson  MJ, El Khoury  J,  et al.  Neuroinflammation in Alzheimer’s disease.  Lancet Neurol. 2015;14(4):388-405. doi:10.1016/S1474-4422(15)70016-5PubMedGoogle ScholarCrossref
45.
Chang  CC, Chow  CC, Tellier  LCAM, Vattikuti  S, Purcell  SM, Lee  JJ.  Second-generation PLINK: rising to the challenge of larger and richer datasets.  Gigascience. 2015;4(1):7. doi:10.1186/s13742-015-0047-8PubMedGoogle ScholarCrossref
46.
Shao  Y, Gan  Z, Epifanovsky  E, et al.  Advances in molecular quantum chemistry contained in the Q-Chem 4 program package.  Molecular Physics. 2015;113(2):184-215. doi:10.1080/00268976.2014.952696Google ScholarCrossref
47.
Fogarty International Center. Collins says global health is one of his top priorities. Global Health Matters Newsletter. July/August 2009;8(4). https://www.fic.nih.gov/News/GlobalHealthMatters/Pages/0809_collins.aspx. Accessed April 22, 2019.
48.
Glänzel  W, de Lange  C.  A distributional approach to multinationality measures of international scientific collaboration.  Scientometrics. 2002;54(1):75-89. doi:10.1023/A:1015684505035Google ScholarCrossref
49.
Katz  JS, Hicks  D.  How much is a collaboration worth? a calibrated bibliometric model.  Scientometrics. 1997;40(3):541-554. doi:10.1007/BF02459299Google ScholarCrossref
50.
Bordons  M, Aparicio  J, Costas  R.  Heterogeneity of collaboration and its relationship with research impact in a biomedical field.  Scientometrics. 2013;96(2):443-466. doi:10.1007/s11192-012-0890-7Google ScholarCrossref
51.
González-Alcaide  G.  Autorías, colaboración y patrones de citación de las revistas biomédicas editadas en España incluidas en el Journal Citation Reports (2003-2007).  Rev Esp Doc Cient. 2010;33(3):397-427. doi:10.3989/redc.2010.3.747Google ScholarCrossref
52.
Abramo  G, D’Angelo  CA, Di Costa  F.  Research collaboration and productivity: is there correlation?  High Educ. 2008;57(2):155-171. doi:10.1007/s10734-008-9139-zGoogle ScholarCrossref
53.
Basu  A, Aggarwal  R.  International collaboration in science in India and its impact on institutional performance.  Scientometrics. 2001;52(3):379-394. doi:10.1023/A:1014239731175Google ScholarCrossref
54.
Kokol  P, Vošner  HBPP.  Discrepancies among Scopus, Web of Science, and PubMed coverage of funding information in medical journal articles.  J Med Libr Assoc. 2018;106(1):81-86. doi:10.5195/JMLA.2018.181PubMedGoogle ScholarCrossref
55.
Franceschini  F, Maisano  D, Mastrogiacomo  L.  Empirical analysis and classification of database errors in Scopus and Web of Science.  J Informetrics. 2016;10(4):933-953. doi:10.1016/j.joi.2016.07.003Google ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    Global Health
    November 22, 2019

    Coauthor Country Affiliations in International Collaborative Research Funded by the US National Institutes of Health, 2009 to 2017

    Author Affiliations
    • 1Fogarty International Center, National Institutes of Health, Bethesda, Maryland
    JAMA Netw Open. 2019;2(11):e1915989. doi:10.1001/jamanetworkopen.2019.15989
    Key Points español 中文 (chinese)

    Question  What are the trends, scope, and importance of US National Institutes of Health–funded publications with international collaboration from 2009 to 2017?

    Findings  In this cross-sectional bibliometric analysis, National Institutes of Health–funded publications indexed in the Web of Science database increased by 46.2%, and the percentage of those with at least 1 coauthor with a non-US affiliation increased from 28.3% to 34.8% from 2009 to 2017. Publications with US-affiliated and non–US-affiliated coauthors had a greater mean citation impact score than those with exclusively US or non-US authors.

    Meaning  International collaboration in National Institutes of Health–funded research is a critical, growing, and high-importance component of the National Institutes of Health’s biomedical and behavioral research mission, contributing to its overall productivity.

    Abstract

    Importance  The US National Institutes of Health (NIH) is the largest funder of biomedical and behavioral research in the world. International collaborative research—a subset of NIH’s portfolio—is critical to furthering the agency’s health research mission.

    Objective  To quantify the extent of the NIH’s international collaborations and the relative importance of this research through the lens of publications.

    Design, Setting, and Participants  This cross-sectional study used bibliometric data from the Web of Science database to analyze trends in the growth of NIH-funded publications from January 1, 2009, to December 31, 2017, and examined their importance using national affiliations of all coauthors listed, h indices, and citation impact scores. All countries with coauthor affiliations in NIH-funded publications during the study period were included. Data were analyzed from October 22 through November 16, 2018.

    Exposures  Country affiliations of coauthors’ institutions in NIH-funded publications indexed in the Web of Science database from 2009 to 2017.

    Main Outcomes and Measures  Trends in the number of NIH-supported publications with non-US coauthors during a 9-year period and their relative importance assessed by h index per country and category-normalized citation impact (CNCI) for groups of country affiliations in 2017.

    Results  From 2009 to 2017, the annual count of NIH-funded publications increased 46.2% from 67 041 to 98 002. This increase was driven in part by an increase in publications with a non-US author alone or as a collaborator with a US author compared with those exclusively with US authors, reflected by an increase in the percentage of publications with non-US coauthors from 28.3% to 34.8%. Moreover, in 2017, publications coauthored by US-affiliated and non–US-affiliated investigators had a higher mean CNCI (1.99) than those whose authors were only US affiliated (1.54) or non–US affiliated (1.35). China became the most frequent publishing partner, with 6982 coauthored publications and the greatest increase over time among non-US countries.

    Conclusions and Relevance  In a 9-year period when the NIH budget remained relatively unchanged, an increase in the number of publications occurred with a growing trend toward more international collaborations of authorship; these publications also had a higher CNCI than publications with only US or only non-US authors. The findings suggest that international collaboration is a vital and growing component of the NIH’s research output and likely reflects increased globalization of biomedical research.

    Introduction

    The US National Institutes of Health (NIH) is the largest funder of biomedical research in the world.1,2 During fiscal year 2019, the NIH used its $39 billion budget to support scientific research in pursuit of improving human health and basic science.3 The NIH funds research at non-US and domestic institutions; much non–US-affiliated research is conducted with funding awarded to US investigators who collaborate with non–US-affiliated scientists or trainees.4 The NIH recognizes that science knows no borders and values the contributions of international collaboration for quality biomedical research. Supporting projects that bring together people and institutions from across the globe provides critical opportunities to advance science, identify new diseases with unique populations, and develop better diagnostics, treatments, and cures.5-7 Clinical trials for infectious diseases, such as human immunodeficiency virus, Ebola virus, dengue, and Zika virus, for instance, require scientists in the United States to work with investigators in affected regions. Particular scientific opportunities that benefit researchers around the world may exist in isolation, such as differences in the response to chemotherapy for breast cancer based on country of origin and genetic background; high prevalence of unique, single-gene alterations that predispose a localized population in Colombia to hereditary, early-onset Alzheimer disease; or interventions to reduce arsenic exposure and its consequences in Bangladesh.8-11 Rapid advancements in communication technology and the trend toward team-based research have facilitated such international collaborations, allowing scientists to freely exchange data and ideas.12,13

    To assess the importance of international research at the NIH through one lens of scientific productivity—peer-reviewed publications—we evaluated the extent of NIH-funded international collaborative research and characterized trends during the past 9 years. Bibliometric analysis offers a powerful tool to better understand this output of research and provides an alternative to other traditional methods for portfolio analysis.14-17 By comparing trends in the number of publications over time and a metric of scores, such as the h index, researchers can describe the relative importance of publications with coauthors affiliated with different countries. To answer larger questions about the importance of US and non-US coauthorship overall, the category-normalized citation impact (CNCI; a normalized metric that describes how frequently an article has been cited compared to other articles in its discipline)18 adds another dimension to the analysis.

    We have conducted a bibliometric study to assess the growth of NIH-funded publications and characterize the scope of international collaboration supported by the NIH during a 9-year period when the NIH budget increased from approximately $30 billion to $35 billion but experienced an inflation-adjusted decrease.19 We evaluated the CNCI and the number of publications overall as a function of coauthors’ country affiliations to achieve these goals. Insights into the global reach of NIH funding is critical to understand the current research landscape and inform future investments. Collaborating institutions, including those from academia, government, and the private sector, can use these findings to guide their own decision-making to build productive international research partnerships.

    Methods

    The Web of Science (WOS; Clarivate Analytics) database was queried for NIH-funded publications using a comprehensive Boolean search string of the full names and acronyms of all NIH institutes and centers in the “funding text” field. We limited our analyses to the 9-year period from January 1, 2009, to December 31, 2017, because WOS only began indexing funding data in 2008, and records for 2018 were incomplete at the time of this analysis.20 The WOS “analyze results” feature was used to sort records by countries or regions and download aggregate counts of NIH-funded publications by country affiliation of coauthors. Countries were categorized by income level according to World Bank country and lending groups.21 Additional search strings were used to query the WOS database for publications by institutes and centers and create reports on coauthors’ national affiliations as described above, using 2017 as the sample year. Publications with funding from multiple institutes and centers were included in each relevant report. InCites (Clarivate Analytics), a WOS tool, was used to calculate the CNCI for all NIH-funded publications in 2017 grouped by coauthors’ national affiliations (US coauthors only, non-US coauthors only, or US and non-US coauthors).18 The CNCI used baseline citation rates by WOS subject area centered around a score of 1.00, with higher scores representing greater importance. All statistics were descriptive and presented counts and proportions of publications in the sample without reference to population parameters. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline was used in the reporting of this cross-sectional study. This analysis of publicly available publication data did not constitute human subjects research and did not require approval by an institutional review board or informed consent.

    Data were analyzed from October 22 through November 16, 2018. The citation report feature in WOS was used to calculate 1-year h indices for each country affiliated with NIH-funded publications in 2017. The h index is a metric that reflects the number of publications and citation counts in a single number.22 For example, an author’s h index would be 20 if he or she had published at least 20 articles that had been cited at least 20 times each. This measure can also be applied to institutions or countries.22-24 Previous bibliometric studies have found 1-year h indices to be highly correlated with those from larger citation windows, making the statistic a valid tool for comparison.25,26 The data were limited to publications in 2017 to match the time frame for annual publication volume. Countries were ranked by number of publications and h index in 2017. Finally, records for all NIH-funded publications from 2009 to 2017 were sorted by citation count.

    Results

    From 2009 to 2017, the number of NIH-funded publications increased 46.2% from 67 041 to 98 002 (Figure 1). The greatest percentage increases were in publications with US-affiliated and non-US-affiliated authors (increase of 13 392 [82.7%]) compared with those exclusively with non–US-affiliated authors (increase of 1702 [61.5%]). Publications with US-affiliated coauthors alone increased more slowly at a rate of 33.0% (n = 15 863).

    As a result of the differential growth rates, the proportions of each group shifted between 2009 and 2017. The fraction of studies with US and non-US coauthors increased from 24.1% (16 188 of 67 041) to 30.2% (29 580 of 98 002), whereas those exclusively with non-US coauthors increased from 4.1% (2773 of 67 041) to 4.6% (4479 of 98 002). Consequently, the overall percentage of publications with non–US-affiliated coauthors increased from 28.3% (18 961 of 67 041) to 34.8% (34 059 of 98 002). Publications with solely US coauthors accounted for 71.7% (48 080 of 67 041) in 2009 but decreased to 65.2% (63 943 of 98 002) in 2017.

    In 2017, the coauthors most represented in NIH-funded publications were from China, the United Kingdom, Canada, and Germany (Figure 2A). Although all countries experienced increases in their annual publication rates, the greatest increases were from publications with coauthors from China (from 1976 to 6982; 253.3%), the United Kingdom (from 2733 to 5582; 104.2%), Canada (from 2614 to 4260; 63.0%), and Germany (from 2059 to 3780; 83.6%). China was the most frequent country affiliated with publications that included US and non-US authors (n = 6982). With the exception of China, the numbers of authors from low- and middle-income countries (LMICs) all experienced sustained lower growth in the same time (Figure 2B).

    To examine our most significant global partnerships in research, we identified the country affiliations most represented among NIH-funded publications in 2017. These were ranked by the total number of publications and their importance as measured by the h index (Figure 3). Of the top 20 most represented country affiliations, 16 were classified by the World Bank as high-income countries (HICs), 3 as upper middle-income countries, 1 as a lower middle-income country, and none as low-income countries. Of the top 20 LMIC affiliations, 12 were upper middle-income countries, 6 were lower middle-income countries, and 2 were low-income countries (Uganda [301 publications] and Tanzania [151 publications]). Using the h index as the primary metric for ranking the importance of a country’s publications, however, elevated the United Kingdom (h index, 66) and Germany (h index, 59) to the top of the list (Figure 3A). Canada (h index, 56) remained third from the top, and China (h index, 50) moved down to the sixth place by h index. India, South Africa, and Taiwan no longer appeared among the top 20 countries, replaced by Austria (h index, 34), Norway (h index, 31), and Singapore (h index, 31). In other words, the latter countries’ publications had a greater importance by this metric despite totaling fewer publications overall.

    The same analysis was repeated using only LMICs (Figure 3B). The h indices for China and Brazil were 50 and 39, respectively, ranking them as the top 2 LMICs by publication output and h index in 2017. Uganda and Tanzania, the only 2 low-income countries ranked among the top 20 LMICs for publication output, did not fall among the top 20 countries by h index. Similarly, Nigeria did not rank among the top 20 countries in the same group. Russia (h index, 30), Colombia (h index, 22), Iran (h index, 21), and Malaysia (h index, 17) ranked substantially higher in the list by h index.

    Of the top 20 countries ranked by h index, 12 were European countries and 4 were high-income East Asian and Pacific countries as classified by the World Bank.21 Internationally collaborative publications tend to be most productive and have greater h indices and CNCI among European partners and Canada as well as various high-income Asian countries and China. China stood out for its most rapid growth in the same period and as the only LMIC among the top 10 represented countries. Marked increases have occurred in NIH-funded publications among LMICs, especially in Africa and Latin America, although these rates still lag behind rates of the HICs.

    The top 10 most cited NIH-funded publications with 1 or more non–US-affiliated coauthors from 2009 to 2017 all included at least 1 US-affiliated coauthor (Table).27-36 Japan and the United Kingdom were the most represented country affiliations (3 publications); followed by China, Germany, Saudi Arabia, Sweden, and Switzerland (2 publications); and then France, Israel, and Singapore (1 publication). Publication topics primarily covered genomics experimental methods and computational biology.

    Publications with LMIC-affiliated coauthors (with or without US coauthors) had much fewer citations (Table).37-46 Most publications were studies of genomics and the global burden of disease, drawing from LMICs and HICs alike. However, all first authors of the top 10 most cited NIH-funded publications in 2017 were affiliated with HICs. The percentage of LMIC-affiliated authors ranged from 2% (Roadmap Epigenomics Consortium40; GTEx Consortium41) to 31% (Vos et al39), with a median of 11.5% in this sample. Only 1 publication (Uhlén et al38) did not include a US-affiliated coauthor.

    For 2017, we analyzed NIH-funded publications among the 27 institutes and centers at the NIH. Of the 98 002 publications in 2017 that reported funding from the NIH, 42 872 (43.7%) did not indicate which institute or center provided support, and 16 788 (30.5%) included funding from more than 1 institute or center and are counted multiple times. The National Cancer Institute was most often acknowledged for US-affiliated publications (7407 [62.06%]) and publications with non–US-affiliated authors (4505 [37.8%]), followed by the National Institute of Allergy and Infectious Diseases (2315 [45.7%]) and the National Heart, Lung, and Blood Institute (1840 [31.5%]). The Fogarty International Center had the highest percentage of non–US-affiliated publications in 2017 (855 [90.0%]), followed by the National Institute of Allergy and Infectious Diseases (2315 [45.7%]), the National Human Genome Research Institute (421 [44.0%]), and the National Institute of Biomedical Imaging and Bioengineering (551 [43.0%]). All of the institutes and centers at the NIH were represented among the set of publications in 2017 with at least 1 non-US coauthor.

    Publications with US and non-US coauthors had the greatest importance as measured by CNCI (1.99), greater than that of publications with only US coauthors (1.54) or those with only non–US co-authors (1.35). The baseline CNCI rate for all NIH-funded publications in the same year was 1.65, placing reports by US-only and non–US-only coauthors below the mean in the sample.

    Discussion

    From 2009 to 2017, a period when the NIH budget was relatively unchanged, the number of NIH-funded publications nonetheless grew by 46.2%, reflecting a global trend in biomedical research. Much of this growth was attributable to the more rapid increase in the number and percentage of publications with non-US coauthors. Furthermore, publications with US- and non–US-affiliated coauthors had greater importance as measured by the CNCI than publications with only US- or only non–US-affiliated authors. Notably, NIH director Francis Collins, MD, PhD, has consistently voiced his support for greater international collaboration, and all 27 institutes were represented with publications having coauthors.47

    These results reinforce earlier findings that internationally collaborative publications tend to be cited more frequently.48,49 However, the rankings by the CNCI in this study are important for 2 additional reasons: (1) they suggest that the trend for greater importance among internationally collaborative papers can be extended to the country level for biomedical research as a whole rather than by institute or subdiscipline within a country, as is already more thoroughly investigated5,12,50-53; and (2) the CNCI also follows the same broad pattern when applied to publications associated with a given funding source. Bibliometric analysis provides another element for portfolio analysis that can be used to summarize the global expansion of biomedical research.

    Of note, the most cited reports primarily covered genomics, methods, and the global burden of disease. The first 2 topics listed rank high among countries regardless of income, whereas burden of disease studies were most common when the scope is limited to LMIC-affiliated publications. Even among these, the proportion of LMIC-affiliated coauthors did not exceed 31%, suggesting that HICs largely drive the research. Considering the large number of collaborators needed to collect enough data for these investigations, collaboration itself was likely a mediator for the enhanced visibility and high CNCI of these publications.

    The predominant country affiliation for non-US authorship is China when ranked by volume, CNCI, and h index. The dramatic increase in coauthored publications may be the result of increased bilateral programs for postdoctorate trainees and visiting fellows, but a rapid survey of the data does not suggest postdoctorate trainees and visiting fellows are the sole contributors to research. Sampled articles listing Chinese-affiliated coauthors generally described equitable distribution of authorship responsibilities when such statements were present.

    Limitations

    This analysis has several potential shortcomings inherent to the methods used. Databases such as WOS do not index all publications and thus cannot provide a complete data set for bibliometric analysis.14,54 However, of the various databases available, WOS captures the most publications in biomedical research with full funding information (29%) and accordingly was used for this study.54 Furthermore, publication data, including funding sources and coauthor country affiliations for each study, can be missing or inaccurate if entered into the database incorrectly.14,55 All authors in the database had country affiliations listed, but publications not captured by the database could nevertheless limit the internal and external validity of the study. The metrics themselves—h index in particular—must be interpreted in context and with the recognition that the country of publication can influence these statistics.14,24 The analysis also does not account for the balance or authorship order of US-affiliated and non–US-affiliated coauthors on a given report. Finally, coauthors’ contributions do not factor into the analysis, eliminating any potential differences among the distribution of those who write the manuscripts, for instance. The distribution of coauthors’ roles by country and country income merits further research. In addition, national affiliation does not necessarily imply different geographical locations for coauthors. International fellows working at NIH-funded facilities in the United States still qualify that publication as an international collaboration if those coauthors maintained their affiliations with non-US institutions. Future research on this topic should consider investigating the different roles and breakdown of authorship by country affiliation. Previous studies have addressed some of these concerns but often on the institutional level rather than on a global scale. Developing a more robust profile of NIH-funded work at this level may help scientists and policy makers support important biomedical research.

    Conclusions

    International collaborative research offers investigators new opportunities to maximize importance and interact with non-US investigators by sharing opportunities from populations having unique features and genetics, exposures, or health care delivery and bringing new technologies to emerging or existing health problems. As biomedical research continues to grow, scientists will need to work together across borders to solve increasingly complex health issues. Improving human health across the globe depends on it.

    Back to top
    Article Information

    Accepted for Publication: October 3, 2019.

    Published: November 22, 2019. doi:10.1001/jamanetworkopen.2019.15989

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2019 Grubbs JC et al. JAMA Network Open.

    Corresponding Author: Peter H. Kilmarx, MD, Fogarty International Center, National Institutes of Health, 31 Center Dr, Mail Stop Code 2220, Bethesda, MD 20892 (peter.kilmarx@nih.gov).

    Author Contributions: Mr Grubbs had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Grubbs, Kilmarx.

    Acquisition, analysis, or interpretation of data: All authors.

    Drafting of the manuscript: Grubbs.

    Critical revision of the manuscript for important intellectual content: All authors.

    Statistical analysis: Grubbs, Glass.

    Administrative, technical, or material support: Grubbs, Kilmarx.

    Supervision: Glass, Kilmarx.

    Conflict of Interest Disclosures: None reported.

    Disclaimer: The findings and conclusions in this study are those of the authors and do not necessarily represent the official position of the National Institutes of Health or US Department of Health and Human Services.

    References
    1.
    National Institutes of Health. Impact of NIH research: our society. https://www.nih.gov/about-nih/what-we-do/impact-nih-research/our-society. Updated May 1, 2018. Accessed November 20, 2018.
    2.
    Viergever  RF, Hendriks  TCC.  The 10 largest public and philanthropic funders of health research in the world: what they fund and how they distribute their funds.  Health Res Policy Syst. 2016;14:12. doi:10.1186/s12961-015-0074-zPubMedGoogle ScholarCrossref
    3.
    National Institutes of Health Office of Budget. National Institutes of Health operating plan for FY 2019. https://officeofbudget.od.nih.gov/pdfs/FY20/cy/FY-2019-NIH-Operating-Plan-2.19.19.pdf. Published February 2018. Accessed November 28, 2018.
    4.
    Heggeness  ML, Evans  L, Pohlhaus  JR, Mills  SL.  Measuring diversity of the National Institutes of Health–funded workforce.  Acad Med. 2016;91(8):1164-1172. doi:10.1097/ACM.0000000000001209PubMedGoogle ScholarCrossref
    5.
    He  ZL, Geng  XS, Campbell-Hunt  C.  Research collaboration and research output: a longitudinal study of 65 biomedical scientists in a New Zealand university.  Res Policy. 2009;38(2):306-317. doi:10.1016/j.respol.2008.11.011Google ScholarCrossref
    6.
    Gazni  A, Didegah  F.  Investigating different types of research collaboration and citation impact: a case study of Harvard University’s publications.  Scientometrics. 2011;87(2):251-265. doi:10.1007/s11192-011-0343-8Google ScholarCrossref
    7.
    Wagner  CS, Staheli  L, Silberglitt  R, Wong  A, Kadtke  J. Linking effectively: learning lessons from successful collaboration in science and technology. RAND Corporation. https://www.rand.org/content/dam/rand/pubs/documented_briefings/2005/DB345.pdf. Published April 2002. Accessed November 28, 2018.
    8.
    Tariot  PN, Lopera  F, Langbaum  JB,  et al; Alzheimer’s Prevention Initiative.  The Alzheimer’s Prevention Initiative Autosomal-Dominant Alzheimer’s Disease Trial: a study of crenezumab versus placebo in preclinical PSEN1 E280A mutation carriers to evaluate efficacy and safety in the treatment of autosomal-dominant Alzheimer’s disease, including a placebo-treated noncarrier cohort.  Alzheimers Dement (N Y). 2018;4:150-160. doi:10.1016/j.trci.2018.02.002PubMedGoogle Scholar
    9.
    Fogarty International Center. Fogarty trainees study Colombian family for clues to prevent Alzheimer’s. Global Health Matters Newsletter. May/June 2017;16(3). https://www.fic.nih.gov/News/Publications/Pages/fogarty-trainees-study-colombian-family-prevent-alzheimers.aspx. Accessed November 28, 2018.
    10.
    Khan  K, Ahmed  E, Factor-Litvak  P,  et al.  Evaluation of an elementary school-based educational intervention for reducing arsenic exposure in Bangladesh.  Environ Health Perspect. 2015;123(12):1331-1336. doi:10.1289/ehp.1409462PubMedGoogle ScholarCrossref
    11.
    Zheng  Y, Walsh  T, Gulsuner  S,  et al.  Inherited breast cancer in Nigerian women.  J Clin Oncol. 2018;36(28):2820-2825. doi:10.1200/JCO.2018.78.3977PubMedGoogle ScholarCrossref
    12.
    Adams  JD; National Bureau of Economic Research.  Scientific Teams and Institutional Collaborations Evidence From US Universities, 1981-1999. Cambridge, MA: National Bureau of Economic Research; 2004. NBER working paper series working paper 10640. https://papers.nber.org/papers/W10640. Accessed November 28, 2018.
    13.
    Frenken  K, Hoekman  J, Kok  S, Ponds  R, van Oort  F, van Vliet  J. Death of distance in science? a gravity approach to research collaboration. In: Pyka  A, Scharnhorst  A, eds.  Innovation Networks: New Approaches in Modelling and Analyzing. Berlin, Heidelberg, Germany: Springer Berlin Heidelberg; 2009:43-57. doi:10.1007/978-3-540-92267-4_3
    14.
    Rigby  J.  Systematic grant and funding body acknowledgement data for publications: new dimensions and new controversies for research policy and evaluation.  Res Eval. 2011;20(5):365-375. doi:10.3152/095820211X13164389670392Google ScholarCrossref
    15.
    Abramo  G, D’Angelo  CA, Di Costa  F.  National research assessment exercises: a comparison of peer review and bibliometrics rankings.  Scientometrics. 2011;89(3):929-941. doi:10.1007/s11192-011-0459-xGoogle ScholarCrossref
    16.
    Abramo  G, D’Angelo  CA.  Evaluating research: from informed peer review to bibliometrics.  Scientometrics. 2011;87(3):499-514. doi:10.1007/s11192-011-0352-7Google ScholarCrossref
    17.
    National Science Board. Science & engineering indicators 2018. National Science Foundation. https://www.nsf.gov/statistics/state-indicators. Published January 15, 2018. Accessed November 27, 2018.
    18.
    Clarivate Analytics. InCites indicators handbook. http://help.incites.clarivate.com/inCites2Live/8980-TRS/version/default/part/AttachmentData/data/InCites-Indicators-Handbook%20-%20June%202018.pdf. Published June 2018. Accessed November 27, 2018.
    19.
    National Institutes of Health Office of Budget. History of Congressional appropriations, fiscal years 2000-2018. https://officeofbudget.od.nih.gov/pdfs/FY19/Approp%20History%20by%20IC%20FY%202000%20-%20FY%202018.pdf. Published February 2019. Accessed April 22, 2019.
    20.
    Web of Science. Web of Science core collection: funding information. https://clarivate.libguides.com/woscc/funding. Updated September 12, 2019. Accessed October 19, 2019.
    21.
    World Bank. World Bank country and lending groups. https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups. Published June 2018. Accessed December 3, 2018.
    22.
    Hirsch  JE.  An index to quantify an individual’s scientific research output.  Proc Natl Acad Sci U S A. 2005;102(46):16569-16572. doi:10.1073/pnas.0507655102PubMedGoogle ScholarCrossref
    23.
    Jacsó  P.  The h-index for countries in Web of Science and Scopus.  Online Inf Rev. 2009;33(4):831-837. doi:10.1108/14684520910985756Google ScholarCrossref
    24.
    Csajbók  E, Berhidi  A, Vasas  L, Schubert  A.  Hirsch-index for countries based on essential science indicators data.  Scientometrics. 2007;73(1):91-117. doi:10.1007/s11192-007-1859-9Google ScholarCrossref
    25.
    Adams  J.  Early citation counts correlate with accumulated impact.  Scientometrics. 2005;63(3):567-581. doi:10.1007/s11192-005-0228-9Google ScholarCrossref
    26.
    Abramo  G, D’Angelo  CA, Cicero  T.  What is the appropriate length of the publication period over which to assess research performance?  Scientometrics. 2012;93(3):1005-1017. doi:10.1007/s11192-012-0714-9Google ScholarCrossref
    27.
    Hanahan  D, Weinberg  RA.  Hallmarks of cancer: the next generation.  Cell. 2011;144(5):646-674. doi:10.1016/j.cell.2011.02.013PubMedGoogle ScholarCrossref
    28.
    Tamura  K, Peterson  D, Peterson  N, Stecher  G, Nei  M, Kumar  S.  MEGA5: Molecular Evolutionary Genetics Analysis using maximum likelihood, evolutionary distance, and maximum parsimony methods.  Mol Biol Evol. 2011;28(10):2731-2739. doi:10.1093/molbev/msr121PubMedGoogle ScholarCrossref
    29.
    Tamura  K, Stecher  G, Peterson  D, Filipski  A, Kumar  S.  MEGA6: Molecular Evolutionary Genetics Analysis version 6.0.  Mol Biol Evol. 2013;30(12):2725-2729. doi:10.1093/molbev/mst197PubMedGoogle ScholarCrossref
    30.
    Adams  PD, Afonine  PV, Bunkóczi  G,  et al.  PHENIX: a comprehensive Python-based system for macromolecular structure solution.  Acta Crystallogr D Biol Crystallogr. 2010;66(pt 2):213-221. doi:10.1107/S0907444909052925PubMedGoogle ScholarCrossref
    31.
    Li  H, Handsaker  B, Wysoker  A,  et al; 1000 Genome Project Data Processing Subgroup.  The Sequence Alignment/Map format and SAMtools.  Bioinformatics. 2009;25(16):2078-2079. doi:10.1093/bioinformatics/btp352PubMedGoogle ScholarCrossref
    32.
    Schindelin  J, Arganda-Carreras  I, Frise  E,  et al.  Fiji: an open-source platform for biological-image analysis.  Nat Methods. 2012;9(7):676-682. doi:10.1038/nmeth.2019PubMedGoogle ScholarCrossref
    33.
    Ronquist  F, Teslenko  M, van der Mark  P,  et al.  MrBayes 3.2: efficient Bayesian phylogenetic inference and model choice across a large model space.  Syst Biol. 2012;61(3):539-542. doi:10.1093/sysbio/sys029PubMedGoogle ScholarCrossref
    34.
    ENCODE Project Consortium.  An integrated encyclopedia of DNA elements in the human genome.  Nature. 2012;489(7414):57-74. doi:10.1038/nature11247PubMedGoogle ScholarCrossref
    35.
    Kumar  S, Stecher  G, Tamura  K.  MEGA7: Molecular Evolutionary Genetics Analysis version 7.0 for bigger data sets.  Mol Biol Evol. 2016;33(7):1870-1874. doi:10.1093/molbev/msw054PubMedGoogle ScholarCrossref
    36.
    Grabherr  MG, Haas  BJ, Yassour  M,  et al.  Full-length transcriptome assembly from RNA-Seq data without a reference genome.  Nat Biotechnol. 2011;29(7):644-652. doi:10.1038/nbt.1883PubMedGoogle ScholarCrossref
    37.
    Auton  A, Brooks  LD, Durbin  RM,  et al; 1000 Genomes Project Consortium.  A global reference for human genetic variation.  Nature. 2015;526(7571):68-74. doi:10.1038/nature15393PubMedGoogle ScholarCrossref
    38.
    Uhlén  M, Fagerberg  L, Hallström  BM,  et al.  Tissue-based map of the human proteome.  Science. 2015;347(6220):1260419. doi:10.1126/science.1260419PubMedGoogle Scholar
    39.
    Vos  T, Barber  RM, Bell  B,  et al; Global Burden of Disease Study 2013 Collaborators.  Global, regional, and national incidence, prevalence, and years lived with disability for 301 acute and chronic diseases and injuries in 188 countries, 1990-2013: a systematic analysis for the Global Burden of Disease Study 2013.  Lancet. 2015;386(9995):743-800. doi:10.1016/S0140-6736(15)60692-4PubMedGoogle ScholarCrossref
    40.
    Kundaje  A, Meuleman  W, Ernst  J,  et al; Roadmap Epigenomics Consortium.  Integrative analysis of 111 reference human epigenomes.  Nature. 2015;518(7539):317-330. doi:10.1038/nature14248PubMedGoogle ScholarCrossref
    41.
    GTEx Consortium.  The Genotype-Tissue Expression (GTEx) pilot analysis: multitissue gene regulation in humans.  Science. 2015;348(6235):648-660. doi:10.1126/science.1262110PubMedGoogle ScholarCrossref
    42.
    Fitzmaurice  C, Dicker  D, Pain  A,  et al; Global Burden of Disease Cancer Collaboration.  The global burden of cancer 2013.  JAMA Oncol. 2015;1(4):505-527. doi:10.1001/jamaoncol.2015.0735PubMedGoogle ScholarCrossref
    43.
    Lundgren  JD, Babiker  AG, Gordin  F,  et al; INSIGHT START Study Group.  Initiation of antiretroviral therapy in early asymptomatic HIV infection.  N Engl J Med. 2015;373(9):795-807. doi:10.1056/NEJMoa1506816PubMedGoogle ScholarCrossref
    44.
    Heneka  MT, Carson  MJ, El Khoury  J,  et al.  Neuroinflammation in Alzheimer’s disease.  Lancet Neurol. 2015;14(4):388-405. doi:10.1016/S1474-4422(15)70016-5PubMedGoogle ScholarCrossref
    45.
    Chang  CC, Chow  CC, Tellier  LCAM, Vattikuti  S, Purcell  SM, Lee  JJ.  Second-generation PLINK: rising to the challenge of larger and richer datasets.  Gigascience. 2015;4(1):7. doi:10.1186/s13742-015-0047-8PubMedGoogle ScholarCrossref
    46.
    Shao  Y, Gan  Z, Epifanovsky  E, et al.  Advances in molecular quantum chemistry contained in the Q-Chem 4 program package.  Molecular Physics. 2015;113(2):184-215. doi:10.1080/00268976.2014.952696Google ScholarCrossref
    47.
    Fogarty International Center. Collins says global health is one of his top priorities. Global Health Matters Newsletter. July/August 2009;8(4). https://www.fic.nih.gov/News/GlobalHealthMatters/Pages/0809_collins.aspx. Accessed April 22, 2019.
    48.
    Glänzel  W, de Lange  C.  A distributional approach to multinationality measures of international scientific collaboration.  Scientometrics. 2002;54(1):75-89. doi:10.1023/A:1015684505035Google ScholarCrossref
    49.
    Katz  JS, Hicks  D.  How much is a collaboration worth? a calibrated bibliometric model.  Scientometrics. 1997;40(3):541-554. doi:10.1007/BF02459299Google ScholarCrossref
    50.
    Bordons  M, Aparicio  J, Costas  R.  Heterogeneity of collaboration and its relationship with research impact in a biomedical field.  Scientometrics. 2013;96(2):443-466. doi:10.1007/s11192-012-0890-7Google ScholarCrossref
    51.
    González-Alcaide  G.  Autorías, colaboración y patrones de citación de las revistas biomédicas editadas en España incluidas en el Journal Citation Reports (2003-2007).  Rev Esp Doc Cient. 2010;33(3):397-427. doi:10.3989/redc.2010.3.747Google ScholarCrossref
    52.
    Abramo  G, D’Angelo  CA, Di Costa  F.  Research collaboration and productivity: is there correlation?  High Educ. 2008;57(2):155-171. doi:10.1007/s10734-008-9139-zGoogle ScholarCrossref
    53.
    Basu  A, Aggarwal  R.  International collaboration in science in India and its impact on institutional performance.  Scientometrics. 2001;52(3):379-394. doi:10.1023/A:1014239731175Google ScholarCrossref
    54.
    Kokol  P, Vošner  HBPP.  Discrepancies among Scopus, Web of Science, and PubMed coverage of funding information in medical journal articles.  J Med Libr Assoc. 2018;106(1):81-86. doi:10.5195/JMLA.2018.181PubMedGoogle ScholarCrossref
    55.
    Franceschini  F, Maisano  D, Mastrogiacomo  L.  Empirical analysis and classification of database errors in Scopus and Web of Science.  J Informetrics. 2016;10(4):933-953. doi:10.1016/j.joi.2016.07.003Google ScholarCrossref
    ×