Determining If Sex Bias Exists in Human Surgical Clinical Research

Key Points

Question  Does sex bias exist in human surgical clinical research?

Findings  In this bibliometric analysis of research articles published in 5 surgical journals, male and female participants are not included in equal numbers, and data are often not reported or analyzed using sex as an independent variable in human surgical clinical research.

Meaning  Sex bias exists in human surgical clinical research, and this disparity must be addressed to improve individualized evidence-based medicine.

Importance  Sex is a variable that is poorly controlled for in clinical research.

Objectives  To determine if sex bias exists in human surgical clinical research, to determine if data are reported and analyzed using sex as an independent variable, and to identify specialties in which the greatest and least sex biases exist.

Design, Setting, and Participants  For this bibliometric analysis, data were abstracted from 1303 original peer-reviewed articles published from January 1, 2011, through December 31, 2012, in 5 surgery journals.

Main Outcomes and Measures  Study type, location, number and sex of participants, degree of sex matching of included participants, and inclusion of sex-based reporting, statistical analysis, and discussion of data.

Results  Of 2347 articles reviewed, 1668 (71.1%) included human participants. After excluding 365 articles, 1303 remained: 17 (1.3%) included males only, 41 (3.1%) included females only, 1020 (78.3%) included males and females, and 225 (17.3%) did not document the sex of the participants. Although female participants represent more than 50% (n = 57 688 606) of the total number (115 377 213) included, considerable variability existed with the number of male (46 111 818), female (58 805 665), and unspecified (10 459 730) participants included among the journals, between US domestic and international studies, and between single vs multicenter studies. For articles included in the study, 38.1% (497 of 1303) reported these data by sex, 33.2% (432 of 1303) analyzed these data by sex, and 22.9% (299 of 1303) included a discussion of sex-based results. Sex matching of the included participants in the research overall was poor, with 45.2% (589 of 1303) of the studies matching the inclusion of both sexes by 50%. During analysis of the different surgical specialties, a wide variation in sex-based inclusion, matching, and data reporting existed, with colorectal surgery having the best matching of male and female participants and cardiac surgery having the worst.

Conclusions and Relevance  Sex bias exists in human surgical clinical research. Few studies included men and women equally, less than one-third performed data analysis by sex, and there was wide variation in inclusion and matching of the sexes among the specialties and the journals reviewed. Because clinical research is the foundation for evidence-based medicine, it is imperative that this disparity be addressed so that therapies benefit both sexes.

In 1977, the US Food and Drug Administration (FDA) excluded women of “childbearing potential” from participating in clinical research until adequate safety and efficacy information could be developed during animal and early clinical studies.¹ This exclusion, along with perceived challenges with recruiting women, subsequently resulted in the poor inclusion of women in clinical trials.^2,3 In addition, women have been historically regarded as suboptimal research participants because of their estrous cycle and inherent reproductive differences from men, although data exist to refute this assumption.^4,5 Because of the low enrollment of women in clinical trials, the National Institutes of Health (NIH) Revitalization Act was introduced and became a law on June 10, 1993. This law mandated the inclusion of women as participants in clinical research funded by the NIH. However, studies have shown that the inclusion of women in clinical trials has marginally improved.^6-8

The equal inclusion of men and women in clinical trials is important as exemplified by the drug zolpidem tartrate (Ambien), a sleep aid medication. Zolpidem was originally approved by the FDA in 1992. In the New Drug Application submitted to the FDA, the peak drug concentration after administration was 45% higher in females, yet the drug was approved for the same dose in males and females.⁹ In 2001, some females taking zolpidem the night before were involved in motor vehicle crashes the next morning.¹⁰ Investigation revealed that women metabolized the drug more slowly than men.¹¹ This finding led to a label change approved by the FDA on May 14, 2013, recommending that women take half the dose as men.¹² To our knowledge, it is currently unknown how many other drugs may be metabolized differently by sex or work differently in males and females. However, the US Government Accountability Office performed an assessment of adverse events by sex and reported that 8 of 10 drugs removed from the market by the FDA were because of adverse events in women.¹³ Males and females have dissimilar health issues, metabolize and react to some drugs differently, respond to medical devices differently, and can have diverse outcomes after medical interventions.^14-23

Surgical research is not exempt from this problem. We recently demonstrated that sex bias exists in surgical biomedical research.²⁴ Based on the results of this study and pressure from the advocacy community, the NIH announced that sex must be considered a variable in all NIH-funded studies beginning January 2016.^25-27 To our knowledge, it remains unknown if the same sex bias exists in human surgical clinical research. Males and females can have different postoperative outcomes, complication rates, and readmission rates, so it is important to know if this problem of sex bias is pervasive in surgery.^28-32 Adequately controlling for sex as a variable with inclusion, data reporting, and data analysis is important because data derived from clinical research are the foundation for evidence-based medicine. Thus, our objectives were to determine if sex bias exists in human surgical clinical research, to determine if data are reported and analyzed using sex as an independent variable, and to identify specialties in which the greatest and least sex biases exist. We hypothesize that males and females are not included in surgical clinical research in equal numbers, and data are not reported or analyzed using sex as an independent variable.

Data were collected as previously described by Yoon et al.²⁴ All original articles published from January 1, 2011, through December 31, 2012, in the top 5 ranked American nonspecialty surgical journals (as determined by Thomson Reuters’ Journal Citation Reports) that publish internationally and among specialties were reviewed. These journals include Annals of Surgery, American Journal of Surgery, JAMA Surgery, The Journal of Surgical Research, and Surgery. Data for this study were collected manually by 1 of 4 data collectors from previously published online articles from the website of each journal and entered into an Excel spreadsheet. Interabstractor accuracy was assessed through pilot data collection to ensure complete agreement among data collectors before full data collection. Internal quality checks on data collection and accuracy were performed by 2 of us (N.A.M. and M.R.K.). Articles excluded from data abstraction were reviews, letters to the editor, case reports, etc. Of the 2347 original peer-reviewed research articles initially identified, publications were further excluded if they included any animal or cell data or the number of included participants was 0 or not stated. Thus, original peer-reviewed research articles (n = 1303) were included in the study. This study reviewed previously published and deidentified publicly available data and was therefore exempt from institutional board review.

The following data were abstracted: type of research (ie, animal, cell, or human), single or multicenter study, Veterans Affairs study vs non–Veterans Affairs study, randomized clinical trial or nonrandomized clinical trial, domestic or international investigation, whether the article reported a sex-specific disease by anatomical criteria (such as prostate, ovarian, cervical, etc), surgical specialty responsible for each study, the number and sex of each participant studied (if specified), and presence of sex-based data reporting. When stratifying for sex-based reporting, articles were assessed for reporting of data, analysis of data by sex, and inclusion of sex-based results in the Discussion section. Finally, the surgical subspecialty responsible for each study was noted. Specialties in which less than 10 articles were published during the 2-year study were marked as other.

The amount of sex matching for the participants included in each study was calculated. The number of included males and females was considered, and the lower number of participants (male or female) in an individual study was defined as the numerator, and the higher number of participants (male or female) was defined as the denominator for individual studies. The percent matching of males and females included as participants was calculated as the ratio of the numerator divided by the denominator multiplied by 100. For example, a 100% match of males and females would include 50 males and 50 females as participants. The equation would equal 50/50 × 100 = 100%. A 50% matching of male and female participants would include 50 males and 25 females, with the equation equal to 25/50 × 100 = 50%.

χ² Tests were used to examine differences among publications by journal that did or did not state sex, to compare the numbers of males, females, and unspecified sex participants in the studies presented in each journal, between domestic and international publications, and among articles published by the different specialties. The differences in the distribution of sex-matched participants, sex-based reporting, analysis, and discussion of the data by specialty were also assessed using a χ² test. The 95% CIs were computed using the Clopper-Pearson method. Significance was assumed for P < .05. Analyses were conducted using SAS, version 9.4 (SAS Institute Inc).

In total, 2347 publications were reviewed among all 5 surgery journals. Of these articles, 1668 (71.1%) included human participants. A total of 365 publications were excluded because they included animals or cells, reported a sex-specific disease, or did not state the number of participants included in the study. Quiz Ref IDOf the 1303 articles included in the final study (Table), 1078 (82.7%) stated the sex of participants included in the study, whereas 225 (17.3%) did not (Figure 1A). Although this finding was consistent among all 5 journals (Figure 1B), the distribution of these differences was statistically significant. The American Journal of Surgery had the most participants with the sex not stated (100 of 392 [25.5%]), whereas Annals of Surgery had the least (21 of 273 [7.7%]) (P < .001). Of the 1078 articles that stated the sex of the participants, 17 (1.6% [95% CI, 0.9%-2.5%]) were male only, 41 (3.8% [95% CI, 3.8%-5.1%]) were female only, and 1020 (94.6% [95% CI, 93.1%-95.9%]) included males and females (Figure 1C) (P < .001). These data were similar between US domestic and international studies. Of the 85.8% (622 of 771) of US domestic publications that stated the sex of the participants, 10 (1.6% [95% CI, 0.8%-2.7%]) were male only, 27 (4.3% [95% CI, 3.0%-6.0%]) were female only, and 585 (94.1% [95% CI, 92.2%-95.7%]) included males and females. Of the 86.2% (456 of 552) international publications that stated the sex of the participants, 7 (1.5% [95% CI, 0.7%-2.9%]) were male only, 14 (3.1% [95% 1.7%-4.8%]) were female only, and 435 (95.4% [95% CI, 93.4%-97.1%]) included males and females.

Quiz Ref IDOf all the 1303 studies included, 38.1% (n = 497) reported data separately for male and female participants, 33.1% (n = 432) performed statistical analysis on data collected by sex, and Quiz Ref ID22.9% (n = 299) of articles addressed sex-based results in the Discussion section (Figure 2). These data were similar between domestic (37%, 32%, and 23%) and international studies (40%, 35%, and 23%, respectively). Among specialties, these data were variable. For example, endocrine surgery, surgical oncology, and colorectal and thoracic surgery were the highest performers in sex-based data reporting (ranging from 46.1% to 50.4%), analysis (40.8% to 48.2%), and discussion (24.8% to 38.9%) of the data (eTable in the Supplement) (P < .001). Breast, bariatric, and cardiac surgery were the lowest performers in sex-based data reporting (ranging from 5.4% to 17.7%), analysis (5.4% to 17.7%), and discussion (1.8% to 15.2%) of the data (P < .001).

In total, during 2 years and among the 5 journals studied, 115 377 213 human participants were included in the published articles. There were 46 111 818 (40%) males, 58 805 665 (51%) females, and 10 459 730 (9%) unspecified participants (Figure 3A). After excluding articles in which military veterans and surgical trainees were the participants, the sex of 114 614 728 participants included (762 485 participants excluded) was similar: 39.7% male, 51.2% female, and 9.1% unspecified. An analysis of published articles on cardiac and thyroid disease, which are more prominent in women, revealed that the sex of 295 147 participants included was 24.6% male, 59% female, and 16.3% unspecified. The total number of males, females, and unspecified participants among journals was variable (Figure 3B). For example, among the 12 424 190 in the American Journal of Surgery, there were 14.5%, 22.3%, and 63.2% of male, female, and unspecified participants, respectively, but for the 13 724 330 participants in the articles in Surgery, there were 46.3%, 51.7%, and 1.9% of male, female, and unspecified participants, respectively (P < .001). Similarly, differences were detected between US domestic and international studies. In US domestic studies, 114 399 214 participants, of which there were 39.9%, 51.2%, and 8.9% of male, female, and unspecified participants, respectively, whereas in international studies with 977 999 participants, there were 48.4%, 26%, and 25.7% of male, female, and unspecified participants, respectively (Figure 3C) (P < .001). In single-center studies with 13 311 832 participants, there were 45.7%, 51.4%, and 2.9% of male, female, and unspecified participants, respectively, whereas in multicenter studies with 102 065 381 participants, there were 39.2%, 50.9%, and 9.9% of male, female, and unspecified participants, respectively (P < .001).

There was inconsistency among surgical subspecialties in the number of male, female, and unspecified participants included in the studies. Quiz Ref IDArticles from trauma and critical care, pediatric, thoracic, and cardiac surgery had greater than 50% (2 168 755 of 4 262 064 trauma and critical care, 79 300 of 129 085 pediatric, 37 316 of 52 175 thoracic, and 30 795 of 42 396 cardiac) male participants (Figure 3D) (P < .001). Quiz Ref IDSurgery unspecified, endocrine, bariatric, and breast surgery had greater than 50% (25 328 490 of 46 805 342 surgery unspecified, 181 302 of 282 089 endocrine, 390 261 of 486 217 bariatric, and 316 526 of 327 371 breast surgery) female participants (Figure 3E) (P < .001). Transplant, general surgery, and articles regarding surgical education and training had greater than 50% (101 640 of 201 209 transplant, 8 020 276 of 11 377 275 general surgery, and 66 773 of 107 509 surgical education and training) of unspecified participants (Figure 3F) (P < .001). Furthermore, there was a significant difference in the overall distribution of male, female, and unspecified participants by specialty (P < .001).

Of all 1303 articles included, 25 (2%) matched the sex of participants by 100%, and 589 (45%) matched the sex of participants by 50% (Figure 4A). In comparing specialties, breast and cardiac surgery did not match any studies by 100%. Examination of the specialties that matched inclusion of males and females by at least 50% revealed that colorectal, transplant, pediatric surgery, and surgical oncology contained the highest number of studies, whereas breast, cardiac, bariatric, and surgical education contained the lowest number of studies. There was a significant difference in the overall distribution of 50% or greater matching among all specialties (Figure 4B).

We show that significant sex bias exists in human surgical clinical research. Most important, of the articles reviewed, approximately one-third statistically analyzed and reported the data by sex. These data were consistent in comparing domestic and international studies and after excluding surgical education and Department of Veterans Affairs studies, the latter of which is predominantly male. Furthermore, the sex of the participants included in the research was not stated in 17.3% (225 of 1303) of published peer-reviewed studies. Of those articles that stated the sex of the participants, 94.6% (1020 of 1078) included males and females. Although female participants represent more than 50% of the total number of those included in the clinical research studies, considerable variability existed with the number of male, female, and unspecified participants included among the journals, between US domestic and international studies, and between single vs multicenter studies. The proportions of male, female, and unspecified participants were unchanged with the exclusion of studies conducted at the Department of Veterans Affairs and on surgical trainees. There was also wide variability in the number of male, female, and unspecified participants included among the surgical specialties. Finally, sex matching of the participants included in the research is practiced in less than half the peer-reviewed publications analyzed using a liberal 50% matching criterion. Thus, the results of this study confirm our hypothesis that males and females are not included in human surgical clinical research in equal numbers, and data are analyzed and discussed using sex as an independent variable in less than one-third of surgical clinical research studies.

To our knowledge, this is the largest and most comprehensive study to examine sex bias in human surgical clinical research. However, there are a few studies in other disciplines that examine sex bias in clinical research. The data presented in our study show an improvement compared with those published by Kwiatkowski et al³³ who revealed that in cancer trials, more men (59.8%) were included than women. Meinert et al⁶ showed that for clinical trials published in US journals from 1966 to 1998, the percentage of participants by sex included males and females (55.2%), males (12.2%), females (11.4%), and unspecified sex (21%). Vidaver et al³⁴ surveyed the medical literature from 1995 to 2000 and showed that less than 20% to 30% (depending on the year) of the studies analyzed data by sex. Similarly, Blauwet et al⁸ disclosed that sex-specific reporting of data was 37.1% in general medical journals and 23% in cardiovascular journals, whereas Geller et al³⁵ revealed that outcomes were not reported by sex in 75% of federally funded randomized clinical trials published in 2009. Finally, in a study of the orthopedic literature, Hettrich et al³⁶ showed an increase in sex-specific analysis from 19.4% to 30.2% from 2000 to 2010. These data are remarkable given that many diseases have a female prevalence.^37-43 Of note, our results are consistent with the US Government Accountability Office report⁴⁴ to the NIH in October 2015, which revealed that more females than males were included in NIH-funded clinical research from 2004 to 2015. In addition, our data regarding the lack of sex-based reporting and analysis are consistent with the US Government Accountability Office’s report and the Institute of Medicine’s 2010 report⁴⁵ on health care research. These later reports showed that despite the NIH Revitalization Act of 1993 and increased female enrollment in clinical trials, sex-based reporting and analysis of results remain areas of disparity.^44,45 Regardless of good overall inclusion of females in human surgical clinical research, we were surprised at the low rate of matching of participants regarding sex. Furthermore, we were amazed that the sex of the participants included was still not reported in more than 17.3% of peer-reviewed studies published in 2011 and 2012. Finally, the wide variation in sex-based data reporting and analysis among surgical specialties was unexpected because some specialties included sex-based data reporting and analysis in more than 50% of published studies, whereas others included them in less than 10%. Together, our data infer that sex disparity exists in human surgical clinical research in many ways despite the government mandate of female inclusion in NIH-funded clinical trials.

Implications of these findings are numerous. First, drugs, therapies, and devices may be developed that are effective for one sex.⁴⁶ Second, for therapies and drugs that have an overall low efficacy in men and women when the data are combined, the therapy or drug may be abandoned; however, that therapy or drug may have greater efficacy in one sex vs the other. This result would be known if sex-based analysis and reporting of the data were performed. For example, the human papillomavirus vaccine is much more effective in women vs men.^47,48 However, collectively, the efficacy is low. If sex-based reporting of the data was not conducted, this therapy that is effective at preventing cervical papillomas in women may not have been developed. Third, therapies may be developed that have undesirable adverse effects in the opposite sex. For example, the odds of an adverse drug reaction in women is 50% greater than in men, women are more likely to be hospitalized because of an adverse drug reaction, and 80% of the drugs removed from the market by the FDA were because of undesirable adverse effects in women.^13,49,50 Thus, whereas it is important to collect data of male and female participants, performing independent data analysis and reporting can produce findings leading to valuable contributions to the health and well-being of males or females independently.

We acknowledge and understand that there are criticisms to including both sexes and considering sex differences in clinical research.^51-54 However, our recommendations are for an FDA mandate that requires drugs, devices, and new therapies be tested equally in male and female participants before market approval. Drugs that have been recalled because of adverse effects in one sex should be tested independently in both sexes because these drugs may be considered for rerelease with different dosing parameters for each sex. Research funding agencies such as the NIH, National Science Foundation, US Department of Defense, the Veterans Affairs Administration, etc, should mandate researchers to match inclusion of males and females in clinical research and report results independently for these participants so that sex may be examined as an independent variable. Journal editors should require authors to include the sex of all participants studied in published literature and require sex-based reporting, analysis, and discussion of data. Supporting these views, the Institute of Medicine published guidelines for sex-specific reporting of research.⁵⁵ Finally, government monitoring of sex-based inclusion of participants, sex as an independent variable, and sex-based data reporting should be required, especially for research conducted using government funds. Such practices have been implemented by other countries, and the United States should do so also.⁵⁶

Limitations exist in this study. Our study design was intended to include a representative sampling of surgical clinical research, not a complete analysis of all surgical clinical research. The intention of this study was limited to articles published in 5 surgery journals during a 2-year period. These journals publish general surgery topics, so there was a lack of articles from certain surgical subspecialties, such as neurological surgery, urology, orthopedic surgery, and plastic surgery. There was no inclusion or specification regarding the difference between sex and intentional use of gender, and this study focused solely on the differences between phenotypic male and female sex.⁵⁷ Furthermore, these are observational data intended to highlight differences in inclusion, reporting, and analysis between sexes and are not corrected for specific disease prevalence. Data inclusion for the funding source was not consistent among the articles; thus, we were not able to consistently and reliably discern if the research was funded and if it was funded by the NIH vs industry. Finally, in the articles analyzed for this study, there were few randomized clinical trials to develop meaningful conclusions; therefore, we are conducting a separate study examining sex bias in all clinical trials registered with ClinicalTrials.gov during a defined period.

This study shows that sex bias exists in human surgical clinical research. Few publications included men and women equally, less than one-third performed data analysis by sex, and there was wide variation in inclusion and matching of the sexes between the specialties and journals reviewed. Because clinical research is the foundation for evidence-based medicine, it is imperative that this disparity be addressed because therapies and practice derived from such studies may be specific to one sex.

Accepted for Publication: April 1, 2016.

Corresponding Author: Melina R. Kibbe, MD, Department of Surgery, University of North Carolina at Chapel Hill, 4041 Burnett Womack, 101 Manning Dr, Campus Box 7050, Chapel Hill, NC 27599 (melina_kibbe@med.unc.edu).

Published Online: August 17, 2016. doi:10.1001/jamasurg.2016.2032

Author Contributions: Drs Mansukhani and Kibbe had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Mansukhani, Yoon, Stubbs, Helenowski, Woodruff, Kibbe.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Mansukhani, Yoon, Teter, Woodruff.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Mansukhani, Yoon, Helenowski.

Obtained funding: Mansukhani.

Administrative, technical, or material support: Mansukhani, Stubbs, Woodruff.

Study supervision: Woodruff, Kibbe.

Conflict of Interest Disclosures: None reported.

Funding/Support: Dr Mansukhani is partially supported by grant 2T32HL094293-06 from the National Institutes of Health.

Role of the Funder/Sponsor: The National Institutes of Health in no part influenced the design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, or approval of the manuscript.

Disclaimer: Dr Kibbe is the editor of JAMA Surgery but was not involved in the editorial review or the decision to accept the manuscript for publication.