Determining If Sex Bias Exists in Human Surgical Clinical Research | Health Disparities | JAMA Surgery | JAMA Network
[Skip to Navigation]
Sign In
Figure 1.  Sex of Participants Included
Sex of Participants Included

Number and percentage of articles that state the sex of the human participants included in all articles (A) and percentage by journal (B). The distribution of rates of sex stated and not stated was different among journals (P < .001). C, Percentage of articles that include male-only participants, female-only participants, and male and female participants.

Figure 2.  Sex-Based Data Reporting
Sex-Based Data Reporting

Percentages are shown of articles that reported the data by sex, statistically analyzed the data by sex, and included a discussion of the results by sex.

Figure 3.  Number of Male and Female Participants Included
Number of Male and Female Participants Included

The number and percentage of male, female, and unspecified participants included in all articles (A), percentage by journal (B), and percentage in US domestic and international studies (C). Distribution of included males, females, and unspecified participants among journals and between domestic and international studies was different (P < .001). D, Percentage of male participants included by specialty. E, Percentage of female participants included by specialty. F, Percentage of unspecified participants included by specialty. Distribution of included males, females, and unspecified participants was different by specialty (P < .001).

Figure 4.  Sex Matching of Participants
Sex Matching of Participants

Amount of sex matching in all articles (A) and by specialty (B). There is a statistically significant difference in distribution of 50% or greater matching among all specialties (P < .001).

Table.  Characteristics of All Articles Included Among Journals, Centers, and Specialties
Characteristics of All Articles Included Among Journals, Centers, and Specialties
1.
US Department of Health, Education, and Welfare.  General Considerations for the Clinical Evaluation of Drugs: US FDA Guideline Document. City, State: Publisher; 1977. Publication No. FDA 77-3040.
2.
Kim  AM, Tingen  CM, Woodruff  TK.  Sex bias in trials and treatment must end.  Nature. 2010;465(7299):688-689.PubMedGoogle ScholarCrossref
3.
Woodruff  TK.  Sex, equity, and science.  Proc Natl Acad Sci U S A. 2014;111(14):5063-5064.PubMedGoogle ScholarCrossref
4.
Mogil  JS, Chanda  ML.  The case for the inclusion of female subjects in basic science studies of pain.  Pain. 2005;117(1-2):1-5.PubMedGoogle ScholarCrossref
5.
Hughes  RN.  Sex does matter: comments on the prevalence of male-only investigations of drug effects on rodent behaviour.  Behav Pharmacol. 2007;18(7):583-589.PubMedGoogle ScholarCrossref
6.
Meinert  CL, Gilpin  AK, Unalp  A, Dawson  C.  Gender representation in trials.  Control Clin Trials. 2000;21(5):462-475.PubMedGoogle ScholarCrossref
7.
Pinnow  E, Sharma  P, Parekh  A, Gevorkian  N, Uhl  K.  Increasing participation of women in early phase clinical trials approved by the FDA.  Womens Health Issues. 2009;19(2):89-93.PubMedGoogle ScholarCrossref
8.
Blauwet  LA, Hayes  SN, McManus  D, Redberg  RF, Walsh  MN.  Low rate of sex-specific result reporting in cardiovascular trials.  Mayo Clin Proc. 2007;82(2):166-170.PubMedGoogle ScholarCrossref
9.
Gordin  MD. US Food and Drug Administration New Drug Application 199-08. http://www.accessdata.fda.gov/drugsatfda_docs/nda/pre96/019908_S000_BIOR.pdf. Published 1991. Accessed January 12, 2016.
10.
Logan  BK, Couper  FJ.  Zolpidem and driving impairment.  J Forensic Sci. 2001;46(1):105-110.PubMedGoogle ScholarCrossref
11.
Farkas  RH, Unger  EF, Temple  R.  Zolpidem and driving impairment—identifying persons at risk.  N Engl J Med. 2013;369(8):689-691.PubMedGoogle ScholarCrossref
12.
US Food and Drug Administration. FDA Drug Safety Communication: FDA approves new label changes and dosing for zolpidem products and a recommendation to avoid driving the day after using Ambien CR. http://www.fda.gov/Drugs/DrugSafety/ucm352085.htm. Published 2013. Accessed July 10, 2016.
13.
Heinrich  J.  GAO-01-286R Drugs Withdrawn From Market. Washington, DC: United States General Accounting Office; 2001.
14.
Anderson  GD.  Sex and racial differences in pharmacological response: where is the evidence? Pharmacogenetics, pharmacokinetics, and pharmacodynamics.  J Womens Health (Larchmt). 2005;14(1):19-29.PubMedGoogle ScholarCrossref
15.
Arnold  AP, Lusis  AJ.  Understanding the sexome: measuring and reporting sex differences in gene systems.  Endocrinology. 2012;153(6):2551-2555.PubMedGoogle ScholarCrossref
16.
Ung  CY, Lam  SH, Zhang  X,  et al.  Inverted expression profiles of sex-biased genes in response to toxicant perturbations and diseases.  PLoS One. 2013;8(2):e56668.PubMedGoogle ScholarCrossref
17.
Oza  NM, Baveja  S, Tedrow  U.  Bridging the gender gap in atrial fibrillation.  Expert Rev Cardiovasc Ther. 2015;13(3):317-323.PubMedGoogle ScholarCrossref
18.
Conrotto  F, D’Ascenzo  F, Salizzoni  S,  et al.  A gender based analysis of predictors of all cause death after transcatheter aortic valve implantation.  Am J Cardiol. 2014;114(8):1269-1274.PubMedGoogle ScholarCrossref
19.
Hess  CN, McCoy  LA, Duggirala  HJ,  et al.  Sex-based differences in outcomes after percutaneous coronary intervention for acute myocardial infarction: a report from TRANSLATE-ACS.  J Am Heart Assoc. 2014;3(1):e000523.PubMedGoogle ScholarCrossref
20.
Hiramoto  JS, Katz  R, Ix  JH,  et al; Health ABC study.  Sex differences in the prevalence and clinical outcomes of subclinical peripheral artery disease in the Health, Aging, and Body Composition (Health ABC) study.  Vascular. 2014;22(2):142-148.PubMedGoogle ScholarCrossref
21.
Walsh  MN.  Risk of obstructive sleep apnea: sex matters.  Circulation. 2015;132(14):1305-1306.PubMedGoogle ScholarCrossref
22.
Woodruff  TK, Kibbe  MR, Paller  AS, Turek  FW, Woolley  CS.  Commentary: “leaning in” to support sex differences in basic science and clinical research.  Endocrinology. 2014;155(4):1181-1183.PubMedGoogle ScholarCrossref
23.
Gall  SL, Donnan  G, Dewey  HM,  et al.  Sex differences in presentation, severity, and management of stroke in a population-based study.  Neurology. 2010;74(12):975-981.PubMedGoogle ScholarCrossref
24.
Yoon  DY, Mansukhani  NA, Stubbs  VC, Helenowski  IB, Woodruff  TK, Kibbe  MR.  Sex bias exists in basic science and translational surgical research.  Surgery. 2014;156(3):508-516.PubMedGoogle ScholarCrossref
25.
Woodruff  TK, Green  S, Paller  A,  et al.  Sex-based biomedical research policy needs an implementation plan.  Womens Health (Lond). 2015;11(4):449-452.PubMedGoogle ScholarCrossref
26.
Clayton  JA.  Studying both sexes: a guiding principle for biomedicine.  FASEB J. 2016;30(2):519-524.PubMedGoogle ScholarCrossref
27.
National Institutes of Health Office of Extramural Research. Consideration of sex as a biological variable in NIH-funded research. http://grants.nih.gov/grants/guide/notice-files/NOT-OD-15-102.html. Published 2015. Accessed July 10, 2016.
28.
Wendling  P. Women dogged by unplanned readmissions after aortic surgery. Cardiology News Digital Network. October 5, 2015. http://www.mdedge.com/ecardiologynews/article/103328/interventional-cardiology-surgery/women-dogged-unplanned-readmissions. Accessed July 10, 2016.
29.
Sutton  JM, Wima  K, Wilson  GC,  et al.  Factors associated with 30-day readmission after restorative proctocolectomy with IPAA: a national study.  Dis Colon Rectum. 2014;57(12):1371-1378.PubMedGoogle ScholarCrossref
30.
Moghavem  N, Morrison  D, Ratliff  JK, Hernandez-Boussard  T.  Cranial neurosurgical 30-day readmissions by clinical indication.  J Neurosurg. 2015;123(1):189-197.PubMedGoogle ScholarCrossref
31.
Otto  W, May  M, Fritsche  HM,  et al.  Analysis of sex differences in cancer-specific survival and perioperative mortality following radical cystectomy: results of a large German multicenter study of nearly 2500 patients with urothelial carcinoma of the bladder.  Gend Med. 2012;9(6):481-489.PubMedGoogle ScholarCrossref
32.
Schmid  M, Shariat  SF, Soave  A, Engel  O, Fisch  M, Rink  M.  Contemporary gender-specific outcomes in Germany after radical cystectomy for bladder cancer.  Curr Urol Rep. 2014;15(6):409.PubMedGoogle ScholarCrossref
33.
Kwiatkowski  K, Coe  K, Bailar  JC, Swanson  GM.  Inclusion of minorities and women in cancer clinical trials, a decade later: have we improved?  Cancer. 2013;119(16):2956-2963.PubMedGoogle ScholarCrossref
34.
Vidaver  RM, Lafleur  B, Tong  C, Bradshaw  R, Marts  SA.  Women subjects in NIH-funded clinical research literature: lack of progress in both representation and analysis by sex.  J Womens Health Gend Based Med. 2000;9(5):495-504.PubMedGoogle ScholarCrossref
35.
Geller  SE, Koch  A, Pellettieri  B, Carnes  M.  Inclusion, analysis, and reporting of sex and race/ethnicity in clinical trials: have we made progress?  J Womens Health (Larchmt). 2011;20(3):315-320.PubMedGoogle ScholarCrossref
36.
Hettrich  CM, Hammoud  S, LaMont  LE, Arendt  EA, Hannafin  JA.  Sex-specific analysis of data in high-impact orthopaedic journals: how are we doing?  Clin Orthop Relat Res. 2015;473(12):3700-3704.PubMedGoogle ScholarCrossref
37.
Greenspan  JD, Craft  RM, LeResche  L,  et al; Consensus Working Group of the Sex, Gender, and Pain SIG of the IASP.  Studying sex and gender differences in pain and analgesia: a consensus report.  Pain. 2007;132(suppl 1):S26-S45.PubMedGoogle ScholarCrossref
38.
Jellinger  KA.  The enigma of mixed dementia.  Alzheimers Dement. 2007;3(1):40-53.PubMedGoogle ScholarCrossref
39.
Office on Women’s Health, US Department of Health and Human Services. Lupus fact sheet. https://www.womenshealth.gov/publications/our-publications/fact-sheet/lupus.html. Updated July 16, 2012. Accessed July 10, 2016.
40.
Centers for Disease Control and Prevention (CDC).  Prevalence of doctor-diagnosed arthritis and arthritis-attributable activity limitation—United States, 2010-2012.  MMWR Morb Mortal Wkly Rep. 2013;62(44):869-873.PubMedGoogle Scholar
41.
Markle  JG, Fish  EN.  SeXX matters in immunity.  Trends Immunol. 2014;35(3):97-104.PubMedGoogle ScholarCrossref
42.
Danska  JS.  Sex matters for mechanism.  Sci Transl Med. 2014;6(258):258fs40.PubMedGoogle ScholarCrossref
43.
Cooper  GS, Stroehla  BC.  The epidemiology of autoimmune diseases.  Autoimmun Rev. 2003;2(3):119-125.PubMedGoogle ScholarCrossref
44.
US Government Accountability Office. National Institutes of Health: better oversight needed to help ensure continued progress including women in health research. http://gao.gov/products/GAO-16-13. Published October 22, 2015. Accessed July 10, 2016.
45.
Institute of Medicine (US) Committee on Women’s Health Research.  Women’s Health Research: Progress, Pitfalls, and Promise. Washington, DC: National Academies Press (US); 2010. http://www.ncbi.nlm.nih.gov/books/NBK210136. Accessed July 17, 2016.
46.
Check Hayden  E.  Sex bias blights drug studies.  Nature. 2010;464(7287):332-333.PubMedGoogle ScholarCrossref
47.
Joura  EA, Giuliano  AR, Iversen  OE,  et al; Broad Spectrum HPV Vaccine Study.  A 9-valent HPV vaccine against infection and intraepithelial neoplasia in women.  N Engl J Med. 2015;372(8):711-723.PubMedGoogle ScholarCrossref
48.
Perkins  RB, Legler  A, Hanchate  A.  Trends in male and female genital warts among adolescents in a safety-net health care system 2004-2013: correlation with introduction of female and male human papillomavirus vaccination.  Sex Transm Dis. 2015;42(12):665-668.PubMedGoogle ScholarCrossref
49.
Zopf  Y, Rabe  C, Neubert  A,  et al.  Women encounter ADRs more often than do men.  Eur J Clin Pharmacol. 2008;64(10):999-1004.PubMedGoogle ScholarCrossref
50.
Tharpe  N.  Adverse drug reactions in women’s health care.  J Midwifery Womens Health. 2011;56(3):205-213.PubMedGoogle ScholarCrossref
51.
Bailey  KR.  Reporting of sex-specific results: a statistician’s perspective.  Mayo Clin Proc. 2007;82(2):158.PubMedGoogle ScholarCrossref
52.
Prins  MH, Smits  KM, Smits  LJ.  Methodologic ramifications of paying attention to sex and gender differences in clinical research.  Gend Med. 2007;4(suppl B):S106-S110.PubMedGoogle ScholarCrossref
53.
Bolon  B.  Gender agenda: sex bias can be justified in animal research.  Nature. 2010;466(7302):28.PubMedGoogle ScholarCrossref
54.
Gendered Innovations. Overemphasizing sex differences as a problem. http://genderedinnovations.stanford.edu/terms/overemphasizing.html. Published 2015. Accessed July 10, 2016.
55.
Institute of Medicine (US) Board on Population Health and Public Health Practice.  Sex-Specific Reporting of Scientific Research: A Workshop Summary. Washington, DC: National Academies Press; 2012.
56.
Zeeb  H, Spallek  J, Jahn  I.  Gender agenda: positive steps taken in Germany.  Nature. 2010;466(7304):315.PubMedGoogle ScholarCrossref
57.
Klinge  I.  Gender perspectives in European research.  Pharmacol Res. 2008;58(3-4):183-189.PubMedGoogle ScholarCrossref
Original Investigation
November 2016

Determining If Sex Bias Exists in Human Surgical Clinical Research

Author Affiliations
  • 1Department of Surgery, Northwestern University, Chicago, Illinois
  • 2Department of Obstetrics and Gynecology, Northwestern University, Chicago, Illinois
  • 3Women’s Health Research Institute, Northwestern University, Chicago, Illinois
  • 4Editor, JAMA Surgery
  • 5now with Department of Surgery, University of North Carolina at Chapel Hill
JAMA Surg. 2016;151(11):1022-1030. doi:10.1001/jamasurg.2016.2032
Key Points

Question  Does sex bias exist in human surgical clinical research?

Findings  In this bibliometric analysis of research articles published in 5 surgical journals, male and female participants are not included in equal numbers, and data are often not reported or analyzed using sex as an independent variable in human surgical clinical research.

Meaning  Sex bias exists in human surgical clinical research, and this disparity must be addressed to improve individualized evidence-based medicine.

Abstract

Importance  Sex is a variable that is poorly controlled for in clinical research.

Objectives  To determine if sex bias exists in human surgical clinical research, to determine if data are reported and analyzed using sex as an independent variable, and to identify specialties in which the greatest and least sex biases exist.

Design, Setting, and Participants  For this bibliometric analysis, data were abstracted from 1303 original peer-reviewed articles published from January 1, 2011, through December 31, 2012, in 5 surgery journals.

Main Outcomes and Measures  Study type, location, number and sex of participants, degree of sex matching of included participants, and inclusion of sex-based reporting, statistical analysis, and discussion of data.

Results  Of 2347 articles reviewed, 1668 (71.1%) included human participants. After excluding 365 articles, 1303 remained: 17 (1.3%) included males only, 41 (3.1%) included females only, 1020 (78.3%) included males and females, and 225 (17.3%) did not document the sex of the participants. Although female participants represent more than 50% (n = 57 688 606) of the total number (115 377 213) included, considerable variability existed with the number of male (46 111 818), female (58 805 665), and unspecified (10 459 730) participants included among the journals, between US domestic and international studies, and between single vs multicenter studies. For articles included in the study, 38.1% (497 of 1303) reported these data by sex, 33.2% (432 of 1303) analyzed these data by sex, and 22.9% (299 of 1303) included a discussion of sex-based results. Sex matching of the included participants in the research overall was poor, with 45.2% (589 of 1303) of the studies matching the inclusion of both sexes by 50%. During analysis of the different surgical specialties, a wide variation in sex-based inclusion, matching, and data reporting existed, with colorectal surgery having the best matching of male and female participants and cardiac surgery having the worst.

Conclusions and Relevance  Sex bias exists in human surgical clinical research. Few studies included men and women equally, less than one-third performed data analysis by sex, and there was wide variation in inclusion and matching of the sexes among the specialties and the journals reviewed. Because clinical research is the foundation for evidence-based medicine, it is imperative that this disparity be addressed so that therapies benefit both sexes.

Introduction

In 1977, the US Food and Drug Administration (FDA) excluded women of “childbearing potential” from participating in clinical research until adequate safety and efficacy information could be developed during animal and early clinical studies.1 This exclusion, along with perceived challenges with recruiting women, subsequently resulted in the poor inclusion of women in clinical trials.2,3 In addition, women have been historically regarded as suboptimal research participants because of their estrous cycle and inherent reproductive differences from men, although data exist to refute this assumption.4,5 Because of the low enrollment of women in clinical trials, the National Institutes of Health (NIH) Revitalization Act was introduced and became a law on June 10, 1993. This law mandated the inclusion of women as participants in clinical research funded by the NIH. However, studies have shown that the inclusion of women in clinical trials has marginally improved.6-8

The equal inclusion of men and women in clinical trials is important as exemplified by the drug zolpidem tartrate (Ambien), a sleep aid medication. Zolpidem was originally approved by the FDA in 1992. In the New Drug Application submitted to the FDA, the peak drug concentration after administration was 45% higher in females, yet the drug was approved for the same dose in males and females.9 In 2001, some females taking zolpidem the night before were involved in motor vehicle crashes the next morning.10 Investigation revealed that women metabolized the drug more slowly than men.11 This finding led to a label change approved by the FDA on May 14, 2013, recommending that women take half the dose as men.12 To our knowledge, it is currently unknown how many other drugs may be metabolized differently by sex or work differently in males and females. However, the US Government Accountability Office performed an assessment of adverse events by sex and reported that 8 of 10 drugs removed from the market by the FDA were because of adverse events in women.13 Males and females have dissimilar health issues, metabolize and react to some drugs differently, respond to medical devices differently, and can have diverse outcomes after medical interventions.14-23

Surgical research is not exempt from this problem. We recently demonstrated that sex bias exists in surgical biomedical research.24 Based on the results of this study and pressure from the advocacy community, the NIH announced that sex must be considered a variable in all NIH-funded studies beginning January 2016.25-27 To our knowledge, it remains unknown if the same sex bias exists in human surgical clinical research. Males and females can have different postoperative outcomes, complication rates, and readmission rates, so it is important to know if this problem of sex bias is pervasive in surgery.28-32 Adequately controlling for sex as a variable with inclusion, data reporting, and data analysis is important because data derived from clinical research are the foundation for evidence-based medicine. Thus, our objectives were to determine if sex bias exists in human surgical clinical research, to determine if data are reported and analyzed using sex as an independent variable, and to identify specialties in which the greatest and least sex biases exist. We hypothesize that males and females are not included in surgical clinical research in equal numbers, and data are not reported or analyzed using sex as an independent variable.

Methods
Data Abstraction

Data were collected as previously described by Yoon et al.24 All original articles published from January 1, 2011, through December 31, 2012, in the top 5 ranked American nonspecialty surgical journals (as determined by Thomson Reuters’ Journal Citation Reports) that publish internationally and among specialties were reviewed. These journals include Annals of Surgery, American Journal of Surgery, JAMA Surgery, The Journal of Surgical Research, and Surgery. Data for this study were collected manually by 1 of 4 data collectors from previously published online articles from the website of each journal and entered into an Excel spreadsheet. Interabstractor accuracy was assessed through pilot data collection to ensure complete agreement among data collectors before full data collection. Internal quality checks on data collection and accuracy were performed by 2 of us (N.A.M. and M.R.K.). Articles excluded from data abstraction were reviews, letters to the editor, case reports, etc. Of the 2347 original peer-reviewed research articles initially identified, publications were further excluded if they included any animal or cell data or the number of included participants was 0 or not stated. Thus, original peer-reviewed research articles (n = 1303) were included in the study. This study reviewed previously published and deidentified publicly available data and was therefore exempt from institutional board review.

Variables Abstracted

The following data were abstracted: type of research (ie, animal, cell, or human), single or multicenter study, Veterans Affairs study vs non–Veterans Affairs study, randomized clinical trial or nonrandomized clinical trial, domestic or international investigation, whether the article reported a sex-specific disease by anatomical criteria (such as prostate, ovarian, cervical, etc), surgical specialty responsible for each study, the number and sex of each participant studied (if specified), and presence of sex-based data reporting. When stratifying for sex-based reporting, articles were assessed for reporting of data, analysis of data by sex, and inclusion of sex-based results in the Discussion section. Finally, the surgical subspecialty responsible for each study was noted. Specialties in which less than 10 articles were published during the 2-year study were marked as other.

Matching of Included Participants by Sex

The amount of sex matching for the participants included in each study was calculated. The number of included males and females was considered, and the lower number of participants (male or female) in an individual study was defined as the numerator, and the higher number of participants (male or female) was defined as the denominator for individual studies. The percent matching of males and females included as participants was calculated as the ratio of the numerator divided by the denominator multiplied by 100. For example, a 100% match of males and females would include 50 males and 50 females as participants. The equation would equal 50/50 × 100 = 100%. A 50% matching of male and female participants would include 50 males and 25 females, with the equation equal to 25/50 × 100 = 50%.

Statistical Analysis

χ2 Tests were used to examine differences among publications by journal that did or did not state sex, to compare the numbers of males, females, and unspecified sex participants in the studies presented in each journal, between domestic and international publications, and among articles published by the different specialties. The differences in the distribution of sex-matched participants, sex-based reporting, analysis, and discussion of the data by specialty were also assessed using a χ2 test. The 95% CIs were computed using the Clopper-Pearson method. Significance was assumed for P < .05. Analyses were conducted using SAS, version 9.4 (SAS Institute Inc).

Results
Sex of Human Participants

In total, 2347 publications were reviewed among all 5 surgery journals. Of these articles, 1668 (71.1%) included human participants. A total of 365 publications were excluded because they included animals or cells, reported a sex-specific disease, or did not state the number of participants included in the study. Quiz Ref IDOf the 1303 articles included in the final study (Table), 1078 (82.7%) stated the sex of participants included in the study, whereas 225 (17.3%) did not (Figure 1A). Although this finding was consistent among all 5 journals (Figure 1B), the distribution of these differences was statistically significant. The American Journal of Surgery had the most participants with the sex not stated (100 of 392 [25.5%]), whereas Annals of Surgery had the least (21 of 273 [7.7%]) (P < .001). Of the 1078 articles that stated the sex of the participants, 17 (1.6% [95% CI, 0.9%-2.5%]) were male only, 41 (3.8% [95% CI, 3.8%-5.1%]) were female only, and 1020 (94.6% [95% CI, 93.1%-95.9%]) included males and females (Figure 1C) (P < .001). These data were similar between US domestic and international studies. Of the 85.8% (622 of 771) of US domestic publications that stated the sex of the participants, 10 (1.6% [95% CI, 0.8%-2.7%]) were male only, 27 (4.3% [95% CI, 3.0%-6.0%]) were female only, and 585 (94.1% [95% CI, 92.2%-95.7%]) included males and females. Of the 86.2% (456 of 552) international publications that stated the sex of the participants, 7 (1.5% [95% CI, 0.7%-2.9%]) were male only, 14 (3.1% [95% 1.7%-4.8%]) were female only, and 435 (95.4% [95% CI, 93.4%-97.1%]) included males and females.

Sex-Based Reporting, Analysis, and Discussion of Data

Quiz Ref IDOf all the 1303 studies included, 38.1% (n = 497) reported data separately for male and female participants, 33.1% (n = 432) performed statistical analysis on data collected by sex, and Quiz Ref ID22.9% (n = 299) of articles addressed sex-based results in the Discussion section (Figure 2). These data were similar between domestic (37%, 32%, and 23%) and international studies (40%, 35%, and 23%, respectively). Among specialties, these data were variable. For example, endocrine surgery, surgical oncology, and colorectal and thoracic surgery were the highest performers in sex-based data reporting (ranging from 46.1% to 50.4%), analysis (40.8% to 48.2%), and discussion (24.8% to 38.9%) of the data (eTable in the Supplement) (P < .001). Breast, bariatric, and cardiac surgery were the lowest performers in sex-based data reporting (ranging from 5.4% to 17.7%), analysis (5.4% to 17.7%), and discussion (1.8% to 15.2%) of the data (P < .001).

Number of Male and Female Participants Studied

In total, during 2 years and among the 5 journals studied, 115 377 213 human participants were included in the published articles. There were 46 111 818 (40%) males, 58 805 665 (51%) females, and 10 459 730 (9%) unspecified participants (Figure 3A). After excluding articles in which military veterans and surgical trainees were the participants, the sex of 114 614 728 participants included (762 485 participants excluded) was similar: 39.7% male, 51.2% female, and 9.1% unspecified. An analysis of published articles on cardiac and thyroid disease, which are more prominent in women, revealed that the sex of 295 147 participants included was 24.6% male, 59% female, and 16.3% unspecified. The total number of males, females, and unspecified participants among journals was variable (Figure 3B). For example, among the 12 424 190 in the American Journal of Surgery, there were 14.5%, 22.3%, and 63.2% of male, female, and unspecified participants, respectively, but for the 13 724 330 participants in the articles in Surgery, there were 46.3%, 51.7%, and 1.9% of male, female, and unspecified participants, respectively (P < .001). Similarly, differences were detected between US domestic and international studies. In US domestic studies, 114 399 214 participants, of which there were 39.9%, 51.2%, and 8.9% of male, female, and unspecified participants, respectively, whereas in international studies with 977 999 participants, there were 48.4%, 26%, and 25.7% of male, female, and unspecified participants, respectively (Figure 3C) (P < .001). In single-center studies with 13 311 832 participants, there were 45.7%, 51.4%, and 2.9% of male, female, and unspecified participants, respectively, whereas in multicenter studies with 102 065 381 participants, there were 39.2%, 50.9%, and 9.9% of male, female, and unspecified participants, respectively (P < .001).

Evaluation of Disparity in Inclusion of Participants by Sex Among Surgical Specialties

There was inconsistency among surgical subspecialties in the number of male, female, and unspecified participants included in the studies. Quiz Ref IDArticles from trauma and critical care, pediatric, thoracic, and cardiac surgery had greater than 50% (2 168 755 of 4 262 064 trauma and critical care, 79 300 of 129 085 pediatric, 37 316 of 52 175 thoracic, and 30 795 of 42 396 cardiac) male participants (Figure 3D) (P < .001). Quiz Ref IDSurgery unspecified, endocrine, bariatric, and breast surgery had greater than 50% (25 328 490 of 46 805 342 surgery unspecified, 181 302 of 282 089 endocrine, 390 261 of 486 217 bariatric, and 316 526 of 327 371 breast surgery) female participants (Figure 3E) (P < .001). Transplant, general surgery, and articles regarding surgical education and training had greater than 50% (101 640 of 201 209 transplant, 8 020 276 of 11 377 275 general surgery, and 66 773 of 107 509 surgical education and training) of unspecified participants (Figure 3F) (P < .001). Furthermore, there was a significant difference in the overall distribution of male, female, and unspecified participants by specialty (P < .001).

Matching Included Participants by Sex

Of all 1303 articles included, 25 (2%) matched the sex of participants by 100%, and 589 (45%) matched the sex of participants by 50% (Figure 4A). In comparing specialties, breast and cardiac surgery did not match any studies by 100%. Examination of the specialties that matched inclusion of males and females by at least 50% revealed that colorectal, transplant, pediatric surgery, and surgical oncology contained the highest number of studies, whereas breast, cardiac, bariatric, and surgical education contained the lowest number of studies. There was a significant difference in the overall distribution of 50% or greater matching among all specialties (Figure 4B).

Discussion

We show that significant sex bias exists in human surgical clinical research. Most important, of the articles reviewed, approximately one-third statistically analyzed and reported the data by sex. These data were consistent in comparing domestic and international studies and after excluding surgical education and Department of Veterans Affairs studies, the latter of which is predominantly male. Furthermore, the sex of the participants included in the research was not stated in 17.3% (225 of 1303) of published peer-reviewed studies. Of those articles that stated the sex of the participants, 94.6% (1020 of 1078) included males and females. Although female participants represent more than 50% of the total number of those included in the clinical research studies, considerable variability existed with the number of male, female, and unspecified participants included among the journals, between US domestic and international studies, and between single vs multicenter studies. The proportions of male, female, and unspecified participants were unchanged with the exclusion of studies conducted at the Department of Veterans Affairs and on surgical trainees. There was also wide variability in the number of male, female, and unspecified participants included among the surgical specialties. Finally, sex matching of the participants included in the research is practiced in less than half the peer-reviewed publications analyzed using a liberal 50% matching criterion. Thus, the results of this study confirm our hypothesis that males and females are not included in human surgical clinical research in equal numbers, and data are analyzed and discussed using sex as an independent variable in less than one-third of surgical clinical research studies.

To our knowledge, this is the largest and most comprehensive study to examine sex bias in human surgical clinical research. However, there are a few studies in other disciplines that examine sex bias in clinical research. The data presented in our study show an improvement compared with those published by Kwiatkowski et al33 who revealed that in cancer trials, more men (59.8%) were included than women. Meinert et al6 showed that for clinical trials published in US journals from 1966 to 1998, the percentage of participants by sex included males and females (55.2%), males (12.2%), females (11.4%), and unspecified sex (21%). Vidaver et al34 surveyed the medical literature from 1995 to 2000 and showed that less than 20% to 30% (depending on the year) of the studies analyzed data by sex. Similarly, Blauwet et al8 disclosed that sex-specific reporting of data was 37.1% in general medical journals and 23% in cardiovascular journals, whereas Geller et al35 revealed that outcomes were not reported by sex in 75% of federally funded randomized clinical trials published in 2009. Finally, in a study of the orthopedic literature, Hettrich et al36 showed an increase in sex-specific analysis from 19.4% to 30.2% from 2000 to 2010. These data are remarkable given that many diseases have a female prevalence.37-43 Of note, our results are consistent with the US Government Accountability Office report44 to the NIH in October 2015, which revealed that more females than males were included in NIH-funded clinical research from 2004 to 2015. In addition, our data regarding the lack of sex-based reporting and analysis are consistent with the US Government Accountability Office’s report and the Institute of Medicine’s 2010 report45 on health care research. These later reports showed that despite the NIH Revitalization Act of 1993 and increased female enrollment in clinical trials, sex-based reporting and analysis of results remain areas of disparity.44,45 Regardless of good overall inclusion of females in human surgical clinical research, we were surprised at the low rate of matching of participants regarding sex. Furthermore, we were amazed that the sex of the participants included was still not reported in more than 17.3% of peer-reviewed studies published in 2011 and 2012. Finally, the wide variation in sex-based data reporting and analysis among surgical specialties was unexpected because some specialties included sex-based data reporting and analysis in more than 50% of published studies, whereas others included them in less than 10%. Together, our data infer that sex disparity exists in human surgical clinical research in many ways despite the government mandate of female inclusion in NIH-funded clinical trials.

Implications of these findings are numerous. First, drugs, therapies, and devices may be developed that are effective for one sex.46 Second, for therapies and drugs that have an overall low efficacy in men and women when the data are combined, the therapy or drug may be abandoned; however, that therapy or drug may have greater efficacy in one sex vs the other. This result would be known if sex-based analysis and reporting of the data were performed. For example, the human papillomavirus vaccine is much more effective in women vs men.47,48 However, collectively, the efficacy is low. If sex-based reporting of the data was not conducted, this therapy that is effective at preventing cervical papillomas in women may not have been developed. Third, therapies may be developed that have undesirable adverse effects in the opposite sex. For example, the odds of an adverse drug reaction in women is 50% greater than in men, women are more likely to be hospitalized because of an adverse drug reaction, and 80% of the drugs removed from the market by the FDA were because of undesirable adverse effects in women.13,49,50 Thus, whereas it is important to collect data of male and female participants, performing independent data analysis and reporting can produce findings leading to valuable contributions to the health and well-being of males or females independently.

We acknowledge and understand that there are criticisms to including both sexes and considering sex differences in clinical research.51-54 However, our recommendations are for an FDA mandate that requires drugs, devices, and new therapies be tested equally in male and female participants before market approval. Drugs that have been recalled because of adverse effects in one sex should be tested independently in both sexes because these drugs may be considered for rerelease with different dosing parameters for each sex. Research funding agencies such as the NIH, National Science Foundation, US Department of Defense, the Veterans Affairs Administration, etc, should mandate researchers to match inclusion of males and females in clinical research and report results independently for these participants so that sex may be examined as an independent variable. Journal editors should require authors to include the sex of all participants studied in published literature and require sex-based reporting, analysis, and discussion of data. Supporting these views, the Institute of Medicine published guidelines for sex-specific reporting of research.55 Finally, government monitoring of sex-based inclusion of participants, sex as an independent variable, and sex-based data reporting should be required, especially for research conducted using government funds. Such practices have been implemented by other countries, and the United States should do so also.56

Limitations exist in this study. Our study design was intended to include a representative sampling of surgical clinical research, not a complete analysis of all surgical clinical research. The intention of this study was limited to articles published in 5 surgery journals during a 2-year period. These journals publish general surgery topics, so there was a lack of articles from certain surgical subspecialties, such as neurological surgery, urology, orthopedic surgery, and plastic surgery. There was no inclusion or specification regarding the difference between sex and intentional use of gender, and this study focused solely on the differences between phenotypic male and female sex.57 Furthermore, these are observational data intended to highlight differences in inclusion, reporting, and analysis between sexes and are not corrected for specific disease prevalence. Data inclusion for the funding source was not consistent among the articles; thus, we were not able to consistently and reliably discern if the research was funded and if it was funded by the NIH vs industry. Finally, in the articles analyzed for this study, there were few randomized clinical trials to develop meaningful conclusions; therefore, we are conducting a separate study examining sex bias in all clinical trials registered with ClinicalTrials.gov during a defined period.

Conclusions

This study shows that sex bias exists in human surgical clinical research. Few publications included men and women equally, less than one-third performed data analysis by sex, and there was wide variation in inclusion and matching of the sexes between the specialties and journals reviewed. Because clinical research is the foundation for evidence-based medicine, it is imperative that this disparity be addressed because therapies and practice derived from such studies may be specific to one sex.

Back to top
Article Information

Accepted for Publication: April 1, 2016.

Corresponding Author: Melina R. Kibbe, MD, Department of Surgery, University of North Carolina at Chapel Hill, 4041 Burnett Womack, 101 Manning Dr, Campus Box 7050, Chapel Hill, NC 27599 (melina_kibbe@med.unc.edu).

Published Online: August 17, 2016. doi:10.1001/jamasurg.2016.2032

Author Contributions: Drs Mansukhani and Kibbe had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Mansukhani, Yoon, Stubbs, Helenowski, Woodruff, Kibbe.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Mansukhani, Yoon, Teter, Woodruff.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Mansukhani, Yoon, Helenowski.

Obtained funding: Mansukhani.

Administrative, technical, or material support: Mansukhani, Stubbs, Woodruff.

Study supervision: Woodruff, Kibbe.

Conflict of Interest Disclosures: None reported.

Funding/Support: Dr Mansukhani is partially supported by grant 2T32HL094293-06 from the National Institutes of Health.

Role of the Funder/Sponsor: The National Institutes of Health in no part influenced the design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, or approval of the manuscript.

Disclaimer: Dr Kibbe is the editor of JAMA Surgery but was not involved in the editorial review or the decision to accept the manuscript for publication.

References
1.
US Department of Health, Education, and Welfare.  General Considerations for the Clinical Evaluation of Drugs: US FDA Guideline Document. City, State: Publisher; 1977. Publication No. FDA 77-3040.
2.
Kim  AM, Tingen  CM, Woodruff  TK.  Sex bias in trials and treatment must end.  Nature. 2010;465(7299):688-689.PubMedGoogle ScholarCrossref
3.
Woodruff  TK.  Sex, equity, and science.  Proc Natl Acad Sci U S A. 2014;111(14):5063-5064.PubMedGoogle ScholarCrossref
4.
Mogil  JS, Chanda  ML.  The case for the inclusion of female subjects in basic science studies of pain.  Pain. 2005;117(1-2):1-5.PubMedGoogle ScholarCrossref
5.
Hughes  RN.  Sex does matter: comments on the prevalence of male-only investigations of drug effects on rodent behaviour.  Behav Pharmacol. 2007;18(7):583-589.PubMedGoogle ScholarCrossref
6.
Meinert  CL, Gilpin  AK, Unalp  A, Dawson  C.  Gender representation in trials.  Control Clin Trials. 2000;21(5):462-475.PubMedGoogle ScholarCrossref
7.
Pinnow  E, Sharma  P, Parekh  A, Gevorkian  N, Uhl  K.  Increasing participation of women in early phase clinical trials approved by the FDA.  Womens Health Issues. 2009;19(2):89-93.PubMedGoogle ScholarCrossref
8.
Blauwet  LA, Hayes  SN, McManus  D, Redberg  RF, Walsh  MN.  Low rate of sex-specific result reporting in cardiovascular trials.  Mayo Clin Proc. 2007;82(2):166-170.PubMedGoogle ScholarCrossref
9.
Gordin  MD. US Food and Drug Administration New Drug Application 199-08. http://www.accessdata.fda.gov/drugsatfda_docs/nda/pre96/019908_S000_BIOR.pdf. Published 1991. Accessed January 12, 2016.
10.
Logan  BK, Couper  FJ.  Zolpidem and driving impairment.  J Forensic Sci. 2001;46(1):105-110.PubMedGoogle ScholarCrossref
11.
Farkas  RH, Unger  EF, Temple  R.  Zolpidem and driving impairment—identifying persons at risk.  N Engl J Med. 2013;369(8):689-691.PubMedGoogle ScholarCrossref
12.
US Food and Drug Administration. FDA Drug Safety Communication: FDA approves new label changes and dosing for zolpidem products and a recommendation to avoid driving the day after using Ambien CR. http://www.fda.gov/Drugs/DrugSafety/ucm352085.htm. Published 2013. Accessed July 10, 2016.
13.
Heinrich  J.  GAO-01-286R Drugs Withdrawn From Market. Washington, DC: United States General Accounting Office; 2001.
14.
Anderson  GD.  Sex and racial differences in pharmacological response: where is the evidence? Pharmacogenetics, pharmacokinetics, and pharmacodynamics.  J Womens Health (Larchmt). 2005;14(1):19-29.PubMedGoogle ScholarCrossref
15.
Arnold  AP, Lusis  AJ.  Understanding the sexome: measuring and reporting sex differences in gene systems.  Endocrinology. 2012;153(6):2551-2555.PubMedGoogle ScholarCrossref
16.
Ung  CY, Lam  SH, Zhang  X,  et al.  Inverted expression profiles of sex-biased genes in response to toxicant perturbations and diseases.  PLoS One. 2013;8(2):e56668.PubMedGoogle ScholarCrossref
17.
Oza  NM, Baveja  S, Tedrow  U.  Bridging the gender gap in atrial fibrillation.  Expert Rev Cardiovasc Ther. 2015;13(3):317-323.PubMedGoogle ScholarCrossref
18.
Conrotto  F, D’Ascenzo  F, Salizzoni  S,  et al.  A gender based analysis of predictors of all cause death after transcatheter aortic valve implantation.  Am J Cardiol. 2014;114(8):1269-1274.PubMedGoogle ScholarCrossref
19.
Hess  CN, McCoy  LA, Duggirala  HJ,  et al.  Sex-based differences in outcomes after percutaneous coronary intervention for acute myocardial infarction: a report from TRANSLATE-ACS.  J Am Heart Assoc. 2014;3(1):e000523.PubMedGoogle ScholarCrossref
20.
Hiramoto  JS, Katz  R, Ix  JH,  et al; Health ABC study.  Sex differences in the prevalence and clinical outcomes of subclinical peripheral artery disease in the Health, Aging, and Body Composition (Health ABC) study.  Vascular. 2014;22(2):142-148.PubMedGoogle ScholarCrossref
21.
Walsh  MN.  Risk of obstructive sleep apnea: sex matters.  Circulation. 2015;132(14):1305-1306.PubMedGoogle ScholarCrossref
22.
Woodruff  TK, Kibbe  MR, Paller  AS, Turek  FW, Woolley  CS.  Commentary: “leaning in” to support sex differences in basic science and clinical research.  Endocrinology. 2014;155(4):1181-1183.PubMedGoogle ScholarCrossref
23.
Gall  SL, Donnan  G, Dewey  HM,  et al.  Sex differences in presentation, severity, and management of stroke in a population-based study.  Neurology. 2010;74(12):975-981.PubMedGoogle ScholarCrossref
24.
Yoon  DY, Mansukhani  NA, Stubbs  VC, Helenowski  IB, Woodruff  TK, Kibbe  MR.  Sex bias exists in basic science and translational surgical research.  Surgery. 2014;156(3):508-516.PubMedGoogle ScholarCrossref
25.
Woodruff  TK, Green  S, Paller  A,  et al.  Sex-based biomedical research policy needs an implementation plan.  Womens Health (Lond). 2015;11(4):449-452.PubMedGoogle ScholarCrossref
26.
Clayton  JA.  Studying both sexes: a guiding principle for biomedicine.  FASEB J. 2016;30(2):519-524.PubMedGoogle ScholarCrossref
27.
National Institutes of Health Office of Extramural Research. Consideration of sex as a biological variable in NIH-funded research. http://grants.nih.gov/grants/guide/notice-files/NOT-OD-15-102.html. Published 2015. Accessed July 10, 2016.
28.
Wendling  P. Women dogged by unplanned readmissions after aortic surgery. Cardiology News Digital Network. October 5, 2015. http://www.mdedge.com/ecardiologynews/article/103328/interventional-cardiology-surgery/women-dogged-unplanned-readmissions. Accessed July 10, 2016.
29.
Sutton  JM, Wima  K, Wilson  GC,  et al.  Factors associated with 30-day readmission after restorative proctocolectomy with IPAA: a national study.  Dis Colon Rectum. 2014;57(12):1371-1378.PubMedGoogle ScholarCrossref
30.
Moghavem  N, Morrison  D, Ratliff  JK, Hernandez-Boussard  T.  Cranial neurosurgical 30-day readmissions by clinical indication.  J Neurosurg. 2015;123(1):189-197.PubMedGoogle ScholarCrossref
31.
Otto  W, May  M, Fritsche  HM,  et al.  Analysis of sex differences in cancer-specific survival and perioperative mortality following radical cystectomy: results of a large German multicenter study of nearly 2500 patients with urothelial carcinoma of the bladder.  Gend Med. 2012;9(6):481-489.PubMedGoogle ScholarCrossref
32.
Schmid  M, Shariat  SF, Soave  A, Engel  O, Fisch  M, Rink  M.  Contemporary gender-specific outcomes in Germany after radical cystectomy for bladder cancer.  Curr Urol Rep. 2014;15(6):409.PubMedGoogle ScholarCrossref
33.
Kwiatkowski  K, Coe  K, Bailar  JC, Swanson  GM.  Inclusion of minorities and women in cancer clinical trials, a decade later: have we improved?  Cancer. 2013;119(16):2956-2963.PubMedGoogle ScholarCrossref
34.
Vidaver  RM, Lafleur  B, Tong  C, Bradshaw  R, Marts  SA.  Women subjects in NIH-funded clinical research literature: lack of progress in both representation and analysis by sex.  J Womens Health Gend Based Med. 2000;9(5):495-504.PubMedGoogle ScholarCrossref
35.
Geller  SE, Koch  A, Pellettieri  B, Carnes  M.  Inclusion, analysis, and reporting of sex and race/ethnicity in clinical trials: have we made progress?  J Womens Health (Larchmt). 2011;20(3):315-320.PubMedGoogle ScholarCrossref
36.
Hettrich  CM, Hammoud  S, LaMont  LE, Arendt  EA, Hannafin  JA.  Sex-specific analysis of data in high-impact orthopaedic journals: how are we doing?  Clin Orthop Relat Res. 2015;473(12):3700-3704.PubMedGoogle ScholarCrossref
37.
Greenspan  JD, Craft  RM, LeResche  L,  et al; Consensus Working Group of the Sex, Gender, and Pain SIG of the IASP.  Studying sex and gender differences in pain and analgesia: a consensus report.  Pain. 2007;132(suppl 1):S26-S45.PubMedGoogle ScholarCrossref
38.
Jellinger  KA.  The enigma of mixed dementia.  Alzheimers Dement. 2007;3(1):40-53.PubMedGoogle ScholarCrossref
39.
Office on Women’s Health, US Department of Health and Human Services. Lupus fact sheet. https://www.womenshealth.gov/publications/our-publications/fact-sheet/lupus.html. Updated July 16, 2012. Accessed July 10, 2016.
40.
Centers for Disease Control and Prevention (CDC).  Prevalence of doctor-diagnosed arthritis and arthritis-attributable activity limitation—United States, 2010-2012.  MMWR Morb Mortal Wkly Rep. 2013;62(44):869-873.PubMedGoogle Scholar
41.
Markle  JG, Fish  EN.  SeXX matters in immunity.  Trends Immunol. 2014;35(3):97-104.PubMedGoogle ScholarCrossref
42.
Danska  JS.  Sex matters for mechanism.  Sci Transl Med. 2014;6(258):258fs40.PubMedGoogle ScholarCrossref
43.
Cooper  GS, Stroehla  BC.  The epidemiology of autoimmune diseases.  Autoimmun Rev. 2003;2(3):119-125.PubMedGoogle ScholarCrossref
44.
US Government Accountability Office. National Institutes of Health: better oversight needed to help ensure continued progress including women in health research. http://gao.gov/products/GAO-16-13. Published October 22, 2015. Accessed July 10, 2016.
45.
Institute of Medicine (US) Committee on Women’s Health Research.  Women’s Health Research: Progress, Pitfalls, and Promise. Washington, DC: National Academies Press (US); 2010. http://www.ncbi.nlm.nih.gov/books/NBK210136. Accessed July 17, 2016.
46.
Check Hayden  E.  Sex bias blights drug studies.  Nature. 2010;464(7287):332-333.PubMedGoogle ScholarCrossref
47.
Joura  EA, Giuliano  AR, Iversen  OE,  et al; Broad Spectrum HPV Vaccine Study.  A 9-valent HPV vaccine against infection and intraepithelial neoplasia in women.  N Engl J Med. 2015;372(8):711-723.PubMedGoogle ScholarCrossref
48.
Perkins  RB, Legler  A, Hanchate  A.  Trends in male and female genital warts among adolescents in a safety-net health care system 2004-2013: correlation with introduction of female and male human papillomavirus vaccination.  Sex Transm Dis. 2015;42(12):665-668.PubMedGoogle ScholarCrossref
49.
Zopf  Y, Rabe  C, Neubert  A,  et al.  Women encounter ADRs more often than do men.  Eur J Clin Pharmacol. 2008;64(10):999-1004.PubMedGoogle ScholarCrossref
50.
Tharpe  N.  Adverse drug reactions in women’s health care.  J Midwifery Womens Health. 2011;56(3):205-213.PubMedGoogle ScholarCrossref
51.
Bailey  KR.  Reporting of sex-specific results: a statistician’s perspective.  Mayo Clin Proc. 2007;82(2):158.PubMedGoogle ScholarCrossref
52.
Prins  MH, Smits  KM, Smits  LJ.  Methodologic ramifications of paying attention to sex and gender differences in clinical research.  Gend Med. 2007;4(suppl B):S106-S110.PubMedGoogle ScholarCrossref
53.
Bolon  B.  Gender agenda: sex bias can be justified in animal research.  Nature. 2010;466(7302):28.PubMedGoogle ScholarCrossref
54.
Gendered Innovations. Overemphasizing sex differences as a problem. http://genderedinnovations.stanford.edu/terms/overemphasizing.html. Published 2015. Accessed July 10, 2016.
55.
Institute of Medicine (US) Board on Population Health and Public Health Practice.  Sex-Specific Reporting of Scientific Research: A Workshop Summary. Washington, DC: National Academies Press; 2012.
56.
Zeeb  H, Spallek  J, Jahn  I.  Gender agenda: positive steps taken in Germany.  Nature. 2010;466(7304):315.PubMedGoogle ScholarCrossref
57.
Klinge  I.  Gender perspectives in European research.  Pharmacol Res. 2008;58(3-4):183-189.PubMedGoogle ScholarCrossref
×