[Skip to Navigation]
Sign In
Figure 1.  CHANGES Participant Recruitment Flowchart
CHANGES Participant Recruitment Flowchart

This recruitment flow represents the entire Cognitive Habits and Growth Evaluation Study (CHANGES) cohort, which includes black residents. Of the cohort, 3588 completed the survey; 196 self-identified as black and were not included in this analysis, resulting in 3392 nonblack residents. From this group, 12 did not complete the burnout items (n = 3380) and 15 did not complete the Patient-Reported Outcome Measurement Information System (PROMIS) items (n = 3377). MS4 indicates medical school year 4; R2, second year of residency; and R3, third year of residency.

Figure 2.  Feeling Thermometer (FT) Score Toward Black People
Feeling Thermometer (FT) Score Toward Black People

Significant differences in FT score across levels of distress are seen. The FT score is obtained by moving a slider along a scale from 0 to 100 points, ranging from very cold or unfavorable (lowest score) to very warm or favorable (highest score).

Table 1.  Demographic Characteristics of Cohort of 3392 Nonblack Second-Year Resident Physicians
Demographic Characteristics of Cohort of 3392 Nonblack Second-Year Resident Physicians
Table 2.  Multivariable Analysis to Identify Factors Associated With Explicit and Implicit Biases Among Second-Year Resident Physicians, Cross-sectional Cohorta
Multivariable Analysis to Identify Factors Associated With Explicit and Implicit Biases Among Second-Year Resident Physicians, Cross-sectional Cohorta
Table 3.  Multivariable Analysis to Identify Factors Associated With Explicit Bias Among Third-Year Resident Physicians, Long-term Cohort
Multivariable Analysis to Identify Factors Associated With Explicit Bias Among Third-Year Resident Physicians, Long-term Cohort
1.
Al-Khatib  SM, Fonarow  GC, Hayes  DL,  et al.  Performance measures to promote quality improvement in sudden cardiac arrest prevention and treatment.  Am Heart J. 2013;165(6):862-868. doi:10.1016/j.ahj.2013.02.010PubMedGoogle ScholarCrossref
2.
Kong  MH, Peterson  ED, Fonarow  GC,  et al.  Addressing disparities in sudden cardiac arrest care and the underutilization of effective therapies.  Am Heart J. 2010;160(4):605-618. doi:10.1016/j.ahj.2010.07.011PubMedGoogle ScholarCrossref
3.
Peterson  E, Yancy  CW.  Eliminating racial and ethnic disparities in cardiac care.  N Engl J Med. 2009;360(12):1172-1174. doi:10.1056/NEJMp0810121PubMedGoogle ScholarCrossref
4.
Yancy  CW.  Disparate care for acute myocardial infarction: moving beyond description and targeting interventions.  Circulation. 2014;130(8):632-633. doi:10.1161/CIRCULATIONAHA.114.011482PubMedGoogle ScholarCrossref
5.
Yancy  CW, Wang  TY, Ventura  HO,  et al; credo Advisory Group.  The coalition to reduce racial and ethnic disparities in cardiovascular disease outcomes (credo): why credo matters to cardiologists.  J Am Coll Cardiol. 2011;57(3):245-252. doi:10.1016/j.jacc.2010.09.027PubMedGoogle ScholarCrossref
6.
Dovidio  JF, Fiske  ST.  Under the radar: how unexamined biases in decision-making processes in clinical interactions can contribute to health care disparities.  Am J Public Health. 2012;102(5):945-952. doi:10.2105/AJPH.2011.300601PubMedGoogle ScholarCrossref
7.
Esnaola  NF, Ford  ME.  Racial differences and disparities in cancer care and outcomes: where’s the rub?  Surg Oncol Clin N Am. 2012;21(3):417-437, viii. doi:10.1016/j.soc.2012.03.012PubMedGoogle ScholarCrossref
8.
Mehta  JL, Bursac  Z, Mehta  P,  et al.  Racial disparities in prescriptions for cardioprotective drugs and cardiac outcomes in Veterans Affairs Hospitals.  Am J Cardiol. 2010;105(7):1019-1023. doi:10.1016/j.amjcard.2009.11.031PubMedGoogle ScholarCrossref
9.
Moy  E, Freeman  W.  Federal investments to eliminate racial/ethnic health-care disparities.  Public Health Rep. 2014;129(suppl 2):62-70. doi:10.1177/00333549141291S212PubMedGoogle ScholarCrossref
10.
Saha  S, Korthuis  PT, Cohn  JA, Sharp  VL, Moore  RD, Beach  MC.  Primary care provider cultural competence and racial disparities in HIV care and outcomes.  J Gen Intern Med. 2013;28(5):622-629. doi:10.1007/s11606-012-2298-8PubMedGoogle ScholarCrossref
11.
van Ryn  M, Burgess  DJ, Dovidio  JF,  et al.  The impact of racism on clinician cognition, behavior, and clinical decision-making.  Du Bois Rev. 2011;8(1):199-218. doi:10.1017/S1742058X11000191PubMedGoogle ScholarCrossref
12.
Epstein  RA.  Disparities and discrimination in health care coverage: a critique of the Institute of Medicine study.  Perspect Biol Med. 2005;48(1)(suppl):S26-S41. doi:10.1353/pbm.2005.0023PubMedGoogle ScholarCrossref
13.
Penner  LA, Dovidio  JF, West  TV,  et al.  Aversive racism and medical interactions with black patients: a field study.  J Exp Soc Psychol. 2010;46(2):436-440. doi:10.1016/j.jesp.2009.11.004PubMedGoogle ScholarCrossref
14.
Paradies  Y, Truong  M, Priest  N.  A systematic review of the extent and measurement of healthcare provider racism.  J Gen Intern Med. 2014;29(2):364-387. doi:10.1007/s11606-013-2583-1PubMedGoogle ScholarCrossref
15.
van Ryn  M, Fu  SS.  Paved with good intentions: do public health and human service providers contribute to racial/ethnic disparities in health?  Am J Public Health. 2003;93(2):248-255. doi:10.2105/AJPH.93.2.248PubMedGoogle ScholarCrossref
16.
Dovidio  JF, ten Vergert  M, Stewart  TL,  et al.  Perspective and prejudice: antecedents and mediating mechanisms.  Pers Soc Psychol Bull. 2004;30(12):1537-1549. doi:10.1177/0146167204271177PubMedGoogle ScholarCrossref
17.
Penner  LA, Gaertner  S, Dovidio  JF,  et al.  A social psychological approach to improving the outcomes of racially discordant medical interactions.  J Gen Intern Med. 2013;28(9):1143-1149. doi:10.1007/s11606-013-2339-yPubMedGoogle ScholarCrossref
18.
Blair  IV, Steiner  JF, Fairclough  DL,  et al.  Clinicians’ implicit ethnic/racial bias and perceptions of care among Black and Latino patients.  Ann Fam Med. 2013;11(1):43-52. doi:10.1370/afm.1442PubMedGoogle ScholarCrossref
19.
Green  AR, Carney  DR, Pallin  DJ,  et al.  Implicit bias among physicians and its prediction of thrombolysis decisions for black and white patients.  J Gen Intern Med. 2007;22(9):1231-1238. doi:10.1007/s11606-007-0258-5PubMedGoogle ScholarCrossref
20.
Sabin  J, Nosek  BA, Greenwald  A, Rivara  FP.  Physicians’ implicit and explicit attitudes about race by MD race, ethnicity, and gender.  J Health Care Poor Underserved. 2009;20(3):896-913. doi:10.1353/hpu.0.0185PubMedGoogle ScholarCrossref
21.
Sabin  JA, Rivara  FP, Greenwald  AG.  Physician implicit attitudes and stereotypes about race and quality of medical care.  Med Care. 2008;46(7):678-685. doi:10.1097/MLR.0b013e3181653d58PubMedGoogle ScholarCrossref
22.
Sabin  JA, Greenwald  AG.  The influence of implicit bias on treatment recommendations for 4 common pediatric conditions: pain, urinary tract infection, attention deficit hyperactivity disorder, and asthma.  Am J Public Health. 2012;102(5):988-995. doi:10.2105/AJPH.2011.300621PubMedGoogle ScholarCrossref
23.
Dehon  E, Weiss  N, Jones  J, Faulconer  W, Hinton  E, Sterling  S.  A systematic review of the impact of physician implicit racial bias on clinical decision-making.  Acad Emerg Med. 2017;24(8):895-904. doi:10.1111/acem.13214PubMedGoogle ScholarCrossref
24.
Finucane  TE, Carrese  JA.  Racial bias in presentation of cases.  J Gen Intern Med. 1990;5(2):120-121. doi:10.1007/BF02600511PubMedGoogle ScholarCrossref
25.
Haider  AH, Sexton  J, Sriram  N,  et al.  Association of unconscious race and social class bias with vignette-based clinical assessments by medical students.  JAMA. 2011;306(9):942-951. doi:10.1001/jama.2011.1248PubMedGoogle ScholarCrossref
26.
Smith  WR, Betancourt  JR, Wynia  MK,  et al.  Recommendations for teaching about racial and ethnic disparities in health and health care.  Ann Intern Med. 2007;147(9):654-665. doi:10.7326/0003-4819-147-9-200711060-00010PubMedGoogle ScholarCrossref
27.
Stone  J, Moskowitz  GB.  Non-conscious bias in medical decision-making: what can be done to reduce it?  Med Educ. 2011;45(8):768-776. doi:10.1111/j.1365-2923.2011.04026.xPubMedGoogle ScholarCrossref
28.
Byrne  A, Tanesini  A.  Instilling new habits: addressing implicit bias in healthcare professionals.  Adv Health Sci Educ Theory Pract. 2015;20(5):1255-1262. doi:10.1007/s10459-015-9600-6PubMedGoogle ScholarCrossref
29.
Dyrbye  LN, Burke  SE, Hardeman  RR,  et al.  Association of clinical specialty with symptoms of burnout and career choice regret among us resident physicians [ retracted in: JAMA. 2019;321(12):1220-1221].  JAMA. 2018;320(11):1114-1130. doi:10.1001/jama.2018.12615PubMedGoogle ScholarCrossref
30.
West  CP, Shanafelt  TD, Kolars  JC.  Quality of life, burnout, educational debt, and medical knowledge among internal medicine residents.  JAMA. 2011;306(9):952-960. doi:10.1001/jama.2011.1247PubMedGoogle ScholarCrossref
31.
Thomas  NK.  Resident burnout.  JAMA. 2004;292(23):2880-2889. doi:10.1001/jama.292.23.2880PubMedGoogle ScholarCrossref
32.
Mata  DA, Ramos  MA, Bansal  N,  et al.  Prevalence of depression and depressive symptoms among resident physicians: a systematic review and meta-analysis.  JAMA. 2015;314(22):2373-2383. doi:10.1001/jama.2015.15845PubMedGoogle ScholarCrossref
33.
Dyrbye  L, Shanafelt  T.  A narrative review on burnout experienced by medical students and residents.  Med Educ. 2016;50(1):132-149. doi:10.1111/medu.12927PubMedGoogle ScholarCrossref
34.
Maslach  C, Jackson  SE, Leiter  MP.  Maslach Burnout Inventory. 4th ed. Menlo Park, CA: Mind Garden, Inc; 2016.
35.
Cunningham  BA, Bonham  VL, Sellers  SL, Yeh  HC, Cooper  LA.  Physicians’ anxiety due to uncertainty and use of race in medical decision-making.  Med Care. 2014;52(8):728-733. doi:10.1097/MLR.0000000000000157PubMedGoogle ScholarCrossref
36.
McConnell  MM, Eva  KW.  The role of emotion in the learning and transfer of clinical skills and knowledge.  Acad Med. 2012;87(10):1316-1322. doi:10.1097/ACM.0b013e3182675af2PubMedGoogle ScholarCrossref
37.
Kuhlmann  S, Piel  M, Wolf  OT.  Impaired memory retrieval after psychosocial stress in healthy young men.  J Neurosci. 2005;25(11):2977-2982. doi:10.1523/JNEUROSCI.5139-04.2005PubMedGoogle ScholarCrossref
38.
Smallwood  J, Fitzgerald  A, Miles  LK, Phillips  LH.  Shifting moods, wandering minds: negative moods lead the mind to wander.  Emotion. 2009;9(2):271-276. doi:10.1037/a0014855PubMedGoogle ScholarCrossref
39.
Dasgupta  N, Desteno  D, Williams  LA, Hunsinger  M.  Fanning the flames of prejudice: the influence of specific incidental emotions on implicit prejudice.  Emotion. 2009;9(4):585-591. doi:10.1037/a0015961PubMedGoogle ScholarCrossref
40.
von Elm  E, Altman  DG, Egger  M, Pocock  SJ, Gøtzsche  PC, Vandenbroucke  JP; STROBE Initiative.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.  Ann Intern Med. 2007;147(8):573-577. doi:10.7326/0003-4819-147-8-200710160-00010PubMedGoogle ScholarCrossref
41.
American Association for Public Opinion Research.  Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 9th ed. https://www.aapor.org/AAPOR_Main/media/publications/Standard-Definitions20169theditionfinal.pdf. Revised 2016. Accessed June 20, 2019.
42.
Hardeman  RR, Burgess  D, Phelan  S, Yeazel  M, Nelson  D, van Ryn  M.  Medical student socio-demographic characteristics and attitudes toward patient centered care: do race, socioeconomic status and gender matter? A report from the Medical Student CHANGES study.  Patient Educ Couns. 2015;98(3):350-355. doi:10.1016/j.pec.2014.11.013PubMedGoogle ScholarCrossref
43.
Hardeman  RR, Przedworski  JM, Burke  SE,  et al.  Mental well-being in first year medical students: a comparison by race and gender: a report from the medical student CHANGE study.  J Racial Ethn Health Disparities. 2015;2(3):403-413. doi:10.1007/s40615-015-0087-xPubMedGoogle ScholarCrossref
44.
Phelan  SM, Burgess  DJ, Puhl  R,  et al.  The adverse effect of weight stigma on the well-being of medical students with overweight or obesity: findings from a national survey.  J Gen Intern Med. 2015;30(9):1251-1258. doi:10.1007/s11606-015-3266-xPubMedGoogle ScholarCrossref
45.
Phelan  SM, Dovidio  JF, Puhl  RM,  et al.  Implicit and explicit weight bias in a national sample of 4,732 medical students: the medical student CHANGES study.  Obesity (Silver Spring). 2014;22(4):1201-1208. doi:10.1002/oby.20687PubMedGoogle ScholarCrossref
46.
Phelan  SM, Puhl  RM, Burke  SE,  et al.  The mixed impact of medical school on medical students’ implicit and explicit weight bias.  Med Educ. 2015;49(10):983-992. doi:10.1111/medu.12770PubMedGoogle ScholarCrossref
47.
van Ryn  M, Hardeman  R, Phelan  SM,  et al.  Medical school experiences associated with change in implicit racial bias among 3547 students: a medical student CHANGES study report.  J Gen Intern Med. 2015;30(12):1748-1756. doi:10.1007/s11606-015-3447-7PubMedGoogle ScholarCrossref
48.
van Ryn  M, Hardeman  RR, Phelan  SM,  et al.  Psychosocial predictors of attitudes toward physician empathy in clinical encounters among 4732 1st year medical students: a report from the CHANGES study.  Patient Educ Couns. 2014;96(3):367-375. doi:10.1016/j.pec.2014.06.009PubMedGoogle ScholarCrossref
49.
Laveist  TA, Nuru-Jeter  A.  Is doctor-patient race concordance associated with greater satisfaction with care?  J Health Soc Behav. 2002;43(3):296-306. doi:10.2307/3090205PubMedGoogle ScholarCrossref
50.
Thornton  RLJ, Powe  NR, Roter  D, Cooper  LA.  Patient-physician social concordance, medical visit communication and patients’ perceptions of health care quality.  Patient Educ Couns. 2011;85(3):e201-e208. doi:10.1016/j.pec.2011.07.015PubMedGoogle ScholarCrossref
51.
Shanafelt  TD, Hasan  O, Dyrbye  LN,  et al.  Changes in burnout and satisfaction with work-life balance in physicians and the general US working population between 2011 and 2014.  [Erratum appears in Mayo Clin Proc. 91(2):276].  Mayo Clin Proc. 2015;90(12):1600-1613. doi:10.1016/j.mayocp.2015.08.023PubMedGoogle ScholarCrossref
52.
Shanafelt  TD, Boone  S, Tan  L,  et al.  Burnout and satisfaction with work-life balance among US physicians relative to the general US population.  Arch Intern Med. 2012;172(18):1377-1385. doi:10.1001/archinternmed.2012.3199PubMedGoogle ScholarCrossref
53.
West  CP, Dyrbye  LN, Satele  DV, Sloan  JA, Shanafelt  TD.  Concurrent validity of single-item measures of emotional exhaustion and depersonalization in burnout assessment.  J Gen Intern Med. 2012;27(11):1445-1452. doi:10.1007/s11606-012-2015-7PubMedGoogle ScholarCrossref
54.
West  CP, Dyrbye  LN, Sloan  JA, Shanafelt  TD.  Single item measures of emotional exhaustion and depersonalization are useful for assessing burnout in medical professionals.  J Gen Intern Med. 2009;24(12):1318-1321. doi:10.1007/s11606-009-1129-zPubMedGoogle ScholarCrossref
55.
Pilkonis  PA, Choi  SW, Reise  SP, Stover  AM, Riley  WT, Cella  D; PROMIS Cooperative Group.  Item banks for measuring emotional distress from the Patient-Reported Outcomes Measurement Information System (PROMIS®): depression, anxiety, and anger.  Assessment. 2011;18(3):263-283. doi:10.1177/1073191111411667PubMedGoogle ScholarCrossref
56.
Kroenke  K, Yu  Z, Wu  J, Kean  J, Monahan  PO.  Operating characteristics of PROMIS four-item depression and anxiety scales in primary care patients with chronic pain.  Pain Med. 2014;15(11):1892-1901. doi:10.1111/pme.12537PubMedGoogle ScholarCrossref
57.
Alwin  D.  Feeling thermometers versus 7-point scales: which are better.  Sociol Methods Res. 1997;25:318-340. doi:10.1177/0049124197025003003Google ScholarCrossref
58.
Burke  SE, Dovidio  JF, Perry  SP,  et al.  Informal training experiences and explicit bias against African Americans among medical students.  Soc Psychol Q. 2017;80:65-84. doi:10.1177/0190272516668166Google ScholarCrossref
59.
Greenwald  AG, McGhee  DE, Schwartz  JLK.  Measuring individual differences in implicit cognition: the implicit association test.  J Pers Soc Psychol. 1998;74(6):1464-1480. doi:10.1037/0022-3514.74.6.1464PubMedGoogle ScholarCrossref
60.
Greenwald  AG, Nosek  BA, Banaji  MR.  Understanding and using the implicit association test: I. An improved scoring algorithm.  J Pers Soc Psychol. 2003;85(2):197-216. doi:10.1037/0022-3514.85.2.197PubMedGoogle ScholarCrossref
61.
Chapman  EN, Kaatz  A, Carnes  M.  Physicians and implicit bias: how doctors may unwittingly perpetuate health care disparities.  J Gen Intern Med. 2013;28(11):1504-1510. doi:10.1007/s11606-013-2441-1PubMedGoogle ScholarCrossref
62.
Accreditation Council for Graduate Medical Education. Data Resource Book. 2010-2011. https://www.acgme.org/About-Us/Publications-and-Resources/Graduate-Medical-Education-Data-Resource-Book. Accessed May 21, 2018
63.
Association of American Medical Colleges. Medical student education: debt, costs, and loan repayment fact card. https://store.aamc.org/downloadable/download/sample/sample_id/240/. Accessed May 21, 2018
64.
Pettigrew  TF, Tropp  LR.  A meta-analytic test of intergroup contact theory.  J Pers Soc Psychol. 2006;90(5):751-783. doi:10.1037/0022-3514.90.5.751PubMedGoogle ScholarCrossref
65.
Johnson  KJ, Fredrickson  BL.  “We all look the same to me”: positive emotions eliminate the own-race in face recognition.  Psychol Sci. 2005;16(11):875-881. doi:10.1111/j.1467-9280.2005.01631.xPubMedGoogle ScholarCrossref
66.
Kang  Y, Gray  JR, Dovidio  JF.  The nondiscriminating heart: lovingkindness meditation training decreases implicit intergroup bias.  J Exp Psychol Gen. 2014;143(3):1306-1313. doi:10.1037/a0034150PubMedGoogle ScholarCrossref
67.
Hagiwara  N, Penner  LA, Gonzalez  R,  et al.  Racial attitudes, physician-patient talk time ratio, and adherence in racially discordant medical interactions.  Soc Sci Med. 2013;87:123-131. doi:10.1016/j.socscimed.2013.03.016PubMedGoogle ScholarCrossref
68.
Eggly  S, Barton  E, Winckles  A, Penner  LA, Albrecht  TL.  A disparity of words: racial differences in oncologist-patient communication about clinical trials.  Health Expect. 2015;18(5):1316-1326. doi:10.1111/hex.12108PubMedGoogle ScholarCrossref
69.
Penner  LA, Dovidio  JF, Edmondson  D,  et al.  The experience of discrimination and black-white health disparities in medical care.  J Black Psychol. 2009;35(2). doi:10.1177/0095798409333585PubMedGoogle Scholar
70.
Cooper  LA, Roter  DL, Carson  KA,  et al.  The associations of clinicians’ implicit attitudes about race with medical visit communication and patient ratings of interpersonal care.  Am J Public Health. 2012;102(5):979-987. doi:10.2105/AJPH.2011.300558PubMedGoogle ScholarCrossref
71.
Dovidio  JF, Penner  LA, Albrecht  TL, Norton  WE, Gaertner  SL, Shelton  JN.  Disparities and distrust: the implications of psychological processes for understanding racial disparities in health and health care.  Soc Sci Med. 2008;67(3):478-486. doi:10.1016/j.socscimed.2008.03.019PubMedGoogle ScholarCrossref
72.
Cuffee  YL, Hargraves  JL, Rosal  M,  et al.  Reported racial discrimination, trust in physicians, and medication adherence among inner-city African Americans with hypertension.  Am J Public Health. 2013;103(11):e55-e62. doi:10.2105/AJPH.2013.301554PubMedGoogle ScholarCrossref
73.
Kronish  IM, Diefenbach  MA, Edmondson  DE, Phillips  LA, Fei  K, Horowitz  CR.  Key barriers to medication adherence in survivors of strokes and transient ischemic attacks.  J Gen Intern Med. 2013;28(5):675-682. doi:10.1007/s11606-012-2308-xPubMedGoogle ScholarCrossref
74.
Martin  KD, Roter  DL, Beach  MC, Carson  KA, Cooper  LA.  Physician communication behaviors and trust among black and white patients with hypertension.  Med Care. 2013;51(2):151-157. doi:10.1097/MLR.0b013e31827632a2PubMedGoogle ScholarCrossref
75.
Amy  NK, Aalborg  A, Lyons  P, Keranen  L.  Barriers to routine gynecological cancer screening for White and African-American obese women.  Int J Obes (Lond). 2006;30(1):147-155. doi:10.1038/sj.ijo.0803105PubMedGoogle ScholarCrossref
76.
Penner  LA, Phelan  SM, Earnshaw  V, Albrecht  TL, Dovidio  JF. Patient stigma, medical interactions, and health care disparities: a selective review. In: Major B, Dovidio JF, Link BG, eds. The Oxford Handbook of Stigma, Discrimination, and Health. New York, NY: Oxford University Press; 2017:183-201.
77.
Shanafelt  T, West  CP, Sinsky  C,  et al.  Changes in burnout and satisfaction with work-life integration in physicians and the general US working population between 2011-2017.  Mayo Clin Proc. 2019;pii:S0025-6196(18)30938-8.Google Scholar
78.
Dyrbye  LN, Shanafelt  TD, Sinsky  CA,  et al.  Burnout Among Health Care Professionals: A Call to Explore and Address This Underrecognized Threat to Safe, High-Quality Care. Washington, DC: National Academy of Medicine; 2017.
79.
West  CP, Dyrbye  LN, Erwin  PJ, Shanafelt  TD.  Interventions to prevent and reduce physician burnout: a systematic review and meta-analysis.  Lancet. 2016;388(10057):2272-2281. doi:10.1016/S0140-6736(16)31279-XPubMedGoogle ScholarCrossref
80.
Panagioti  M, Panagopoulou  E, Bower  P,  et al.  Controlled interventions to reduce burnout in physicians: a systematic review and meta-analysis.  JAMA Intern Med. 2017;177(2):195-205. doi:10.1001/jamainternmed.2016.7674PubMedGoogle ScholarCrossref
81.
Cohen  J.  Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Abingdon-on-Thames, UK: Routledge; 1988.
Original Investigation
Medical Education
July 26, 2019

Association of Racial Bias With Burnout Among Resident Physicians

Author Affiliations
  • 1Division of Community Internal Medicine, Department of Medicine, Mayo Clinic, Rochester, Minnesota
  • 2Department of Internal Medicine, Yale School of Medicine, Charlottesville, Virginia
  • 3Division of General Internal Medicine, Department of Medicine, Mayo Clinic, Rochester, Minnesota
  • 4Department of Psychology, Yale University, New Haven, Connecticut
  • 5School of Public Health, Division of Health Policy and Management, University of Minnesota, Minneapolis
  • 6Department of Psychology, Syracuse University, Syracuse, New York
  • 7Division of Health Care Policy and Research, Mayo Clinic, Rochester, Minnesota
  • 8Department of Family Medicine and Community Health, University of Minnesota, Minneapolis
  • 9School of Nursing, Oregon Health and Science University, Portland
JAMA Netw Open. 2019;2(7):e197457. doi:10.1001/jamanetworkopen.2019.7457
Key Points

Question  Are symptoms of burnout associated with resident physicians’ implicit and explicit biases toward black people?

Findings  In this cohort study of 3392 second-year resident physicians who self-identified as nonblack, symptoms of burnout were associated with greater explicit and implicit racial biases. Recovery from burnout in the third year of residency was associated with the greatest reduction in explicit bias toward black people.

Meaning  Given the high prevalence of burnout among resident physicians and the negative association between bias and suboptimal medical care, symptoms of burnout may be factors in disparities in care; the implications for the quality of care provided to black people and other disadvantaged groups could be substantial.

Abstract

Importance  Burnout, a syndrome characterized by emotional exhaustion, depersonalization, and a decreased sense of efficacy, is common among resident physicians, and negative emotional states may increase the expression of prejudices, which are associated with racial disparities in health care. Whether racial bias varies by symptoms of burnout among resident physicians is unknown.

Objective  To assess the association between burnout and explicit and implicit racial biases toward black people in resident physicians.

Design, Setting, and Participants  This cohort study obtained data from surveys completed by first-year medical students and resident physicians in the United States as part of the Cognitive Habits and Growth Evaluation Study. Participants were followed up from enrollment in 2010 to 2011 through 2017. Participants completed questionnaires at year 4 of medical school as well as at the second and third years of residency. Only data from resident physicians who self-identified as belonging to a racial group other than black (n = 3392) were included in the analyses because of scarce evidence of racial bias in the care provided to black patients by black physicians. Resident physicians training in radiology or pathology were excluded because they provided less direct patient interaction.

Main Outcomes and Measures  Burnout symptoms were measured by 2 single-item measures from the Maslach Burnout Inventory. Explicit attitudes about white and black people were measured by a feeling thermometer (FT, from 0 to 100 points, ranging from very cold or unfavorable [lowest score] to very warm or favorable [highest score]; included in the second-year [R2] and third-year [R3] questionnaires). The R2 Questionnaire included a racial Implicit Association Test (IAT; range: –2 to 2).

Results  Among the 3392 nonblack resident physician respondents, 1693 (49.9%) were male, 1964 (57.9%) were younger than 30 years, and 2362 (69.6%) self-identified as belonging to the white race. In this cohort, 1529 of 3380 resident physicians (45.2%) had symptoms of burnout and 1394 of 3377 resident physicians (41.3%) had depression. From this group, 12 did not complete the burnout items and 15 did not complete the Patient-Reported Outcomes Measurement Information System (PROMIS) items. The mean (SD) FT score toward black people was 77.9 (21.0) and toward white people was 81.1 (20.1), and the mean (SD) racial IAT score was 0.4 (0.4). Burnout at the R2 Questionnaire time point was associated with greater explicit and implicit racial biases. In multivariable analyses adjusting for demographics, specialty, depression, and FT scores toward white people, resident physicians with burnout had greater explicit racial bias (difference in FT score, –2.40; 95% CI, –3.42 to –1.37; P < .001) and implicit racial bias (difference in IAT score, 0.05; 95% CI, 0.02-0.08; P = .002). A dose-response association was found between change in depersonalization from R2 to R3 Questionnaire and R3 Questionnaire explicit bias (for each 1-point increase the difference in R3 FT score decreased, –0.73; 95% CI, –1.23 to –0.23; P = .004) and change in explicit bias.

Conclusions and Relevance  Among resident physicians, symptoms of burnout appeared to be associated with greater explicit and implicit racial biases; given the high prevalence of burnout and the negative implications of bias for medical care, symptoms of burnout may be factors in racial disparities in health care.

Introduction

Despite efforts on multiple fronts, substantial morbidity and mortality differences persist between white and black patients, regardless of their socioeconomic status and level of education.1-5 Although multiple complex factors are associated with this racial disparity in health status, the difference in medical care provided by physicians to black patients compared with white patients is a substantial aspect.6-11 Previous studies have found that, although physicians consciously value equitable care,12,13 their directly expressed (explicit) and unconscious (implicit) biases are factors in their behaviors and decisions that are associated with the medical care they actually provide.14-22 These data, coupled with evidence of racial bias among trainees,23-25 have led to calls for graduate medical education to include curricula focused on understanding and addressing racial health disparities.26-28

Burnout is prevalent among resident physicians29-31 and is an underrecognized threat to the success of curricular interventions. A substantial body of literature has documented a high prevalence of burnout and depression among resident physicians.32,33 Burnout, a syndrome characterized by emotional exhaustion, depersonalization (ie, cynicism), and a decreased sense of efficacy, is job related, situation specific, and largely driven by work-related factors.34 Among resident physicians, the primary drivers of burnout include work intensity, suboptimal supervisor behaviors, lack of flexibility and control, educational debt, and work-home conflict.33 Physicians’ negative emotional states have been shown to be associated with greater explicit racial bias in medical decision-making.35 Negative emotions, like those characterized by burnout and depression, also can impede cognitive performance,36-38 making implicit biases more likely to play a role in behaviors and decision-making.39 Burnout, as a negative emotional state, could activate bias, reduce cognitive capacity leading to inappropriate application of heuristics, or have negative consequences in mindful decision making in other ways.

To our knowledge, the potential association between burnout and explicit and implicit racial biases in resident physicians has not been previously studied. Therefore, we undertook a longitudinal study to assess this association in a national sample of resident physicians, who had been followed up since their first year of medical school as participants in the Cognitive Habits and Growth Evaluation Study (CHANGES).

Methods

The institutional review boards of the University of Minnesota, Oregon Health and Sciences University, and Mayo Clinic approved this study. Written informed consent was provided by all CHANGES participants. Methods and results are reported in accordance with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines for cohort and cross-sectional studies40 and with the American Association for Public Opinion Research (AAPOR) guidelines for surveys.41

The methods used in CHANGES have been previously reported.29,42-48 Briefly, in 2010 to 2011, first-year medical students attending a stratified random sample of 49 allopathic US medical schools were invited to participate in CHANGES (Figure 1). In 2014, medical students who had consented and provided baseline data (ie, baseline respondents) were invited to complete the year 4 of medical school questionnaire (MS4 Questionnaire). Subsequently, in 2016, baseline respondents were invited to complete the second year of residency questionnaire (R2 Questionnaire). In 2017, resident physicians who had completed the MS4 Questionnaire and were not training in radiology or pathology (specialties excluded because they provided less direct patient interaction) were invited to complete the third year of residency questionnaire (R3 Questionnaire).

At each time point, written informed consent was provided by the respondent. The last year of follow-up was 2017. Participants received financial compensation for each questionnaire they completed.

We included data from only nonblack resident physicians (ie, individuals who defined themselves as belonging to a racial group other than black). Black resident physicians were excluded because there was less evidence of racial bias in the care provided to black patients by black physicians.20,49,50

Measures

The questionnaires (eAppendix in the Supplement) included questions about demographic characteristics (year of birth, sex, race, ethnicity, relationship status, and parental status), specialty, burnout, depression, and attitudes about black and white people.

Burnout and Depressive Symptoms

Consistent with approaches used in other large studies,29,30,51,52 we measured symptoms of burnout on the R2 Questionnaire and the R3 Questionnaire using 2 single-item measures (a question on emotional exhaustion, and another on depersonalization) adapted from the full 22-item Maslach Burnout Inventory and with 7 levels of responses ranging from never to every day. Previous studies including several independent samples of more than 10 000 physicians and medical students have demonstrated that these 2 single-item measures stratify the risk of burnout.53,54 In these studies, the likelihood ratios for the once-a-week or the more-often response to the emotional exhaustion measure ranged from 6 to 42 and to the depersonalization measure ranged from 16 to 37 compared with the respective scale of the full Maslach Burnout Inventory.53,54 In addition, the positive predictive value of high levels of emotional exhaustion was 88.2% and depersonalization was 89.6%; the area under the receiver operating characteristic curve for the emotional exhaustion measure was 0.94 and for the depersonalization measure was 0.93, in comparison to the full Maslach Burnout Inventory domain scores.53,54

In these previous studies, dichotomized overall burnout (indicated by a high score, defined as having weekly or more often symptoms of either emotional exhaustion or depersonalization) was associated with patient care and physician well-being outcomes, with magnitudes of association similar to those of overall burnout as measured by the full Maslach Burnout Inventory.53,54 In this present study, we used overall burnout based on the single items in our primary analyses. We repeated the same analyses using the emotional exhaustion and depersonalization scales as continuous variables (the findings are reported in eTable 1 in the Supplement). We measured symptoms of depression on the R2 Questionnaire using the depression short form 4a of PROMIS (Patient-Reported Outcomes Measurement Information System), an instrument developed and validated by the National Institutes of Health.55 Respondents were asked to rate how often in the past 7 days they experienced affective and cognitive manifestations of depression. Response options were never, almost never, sometimes, fairly often, and very often with scores ranging from 4 to 20. The area under the receiver operating characteristic curve for detecting individuals with major depression, as diagnosed by the 9-item Patient Health Questionnaire, was 0.90.56 Participants were considered to have depressive symptoms if they had PROMIS depression scores of 8 or higher (sensitivity of 83.1% and specificity of 84.3%).56

Explicit and Implicit Racial Biases

On the R2 and R3 Questionnaires, participants indicated their feelings about black (African American in the survey instrument) individuals on a feeling thermometer (FT), by moving a slider along a scale from 0 to 100 points, ranging from very cold or unfavorable (lowest score) to very warm or favorable (highest score). Respondents answered similar questions regarding their feelings about white (Caucasian in the survey instrument) people. An FT has been established as valid for measuring attitudes about various social groups57 and has been used to assess attitudes toward black people.18,58 Explicit bias about black people was captured by adjusting FT scores toward black people for FT scores toward white people.

The R2 Questionnaire also included an Implicit Association Test (IAT; range: –2 to 2) to measure implicit bias toward black people compared with implicit bias toward white people (a positive score indicates greater prowhite bias).59 During the IAT, participants sorted pictures of people of European and African origin and words (eg, beautiful, cheerful, friend, failure, tragic, and scorn). In one block, participants were instructed to categorize images and words either as white people or good or as black people or bad; in the other block, they were instructed to categorize images and words either as white people or bad or as black people or good. Relative preference for white women and white men over black women and black men (IAT D score) was calculated by subtracting the mean response latency for the former IAT practice trials from the mean response latency for the latter practice trials and then dividing by the SD for all practice trials.60 The IAT score ranges from –2 (strong preference for black men and women) to 2 (strong preference for white women and men). Previous studies have used the race IAT to measure implicit racial bias.19,20,61 The IAT was not included in the R3 Questionnaire.

Statistical Analysis

Data analysis was conducted from March 1, 2018, through December 21, 2018, and then again from April 30, 2019, to May 7, 2019. Response rates at each time point were calculated using standard methods.41 In addition to evaluating basic summary statistics of respondent characteristics, FT scores, and IAT scores, we assessed the differences in FT scores between the R2 and R3 Questionnaires using paired t tests. Using analysis of variance models, we assessed the bivariate association between high emotional exhaustion, high depersonalization, overall burnout (high emotional exhaustion and/or high depersonalization), and symptoms of depression as well as racial implicit and explicit biases at the R2 Questionnaire time point; we reported the mean (SD) bias score for each response group and the overall P value for each categorical variable. For dichotomous variables, we also estimated the mean (95% CI) of the difference in bias scores between the groups. Then, to assess the independent association of burnout and depression with implicit and explicit biases at the R2 Questionnaire time point, we estimated for each bias measure a multivariable regression model that included indicators for burnout and depression as well as age, sex, race, ethnicity, relationship status, parental status, and specialty; we reported coefficients with 95% CIs for each burnout and depression group and overall P values for each variable. The explicit bias in the second year of residency model also included the R2 Questionnaire FT score toward white people as a covariate.

To examine the association of changes in symptoms of burnout with changes in explicit bias, we classified R2 and R3 Questionnaire respondents into chronic burnout (symptoms of burnout at both the second and third years of residency time points), never burned out (did not have symptoms of burnout at either time point), recovered from burnout (had symptoms of burnout at the second but not third year of residency time point), and new burnout (had symptoms of burnout only at the third year of residency time point). We then estimated a regression model in which R3 Questionnaire FT score toward black people was the dependent variable and burnout pattern was the independent variable, adjusting for age, sex, race, ethnicity, relationship status, parental status, specialty, R2 Questionnaire reported depression, R2 Questionnaire FT score toward white people, and R2 Questionnaire FT score toward black people.

In secondary analyses, we replicated the main analyses using the components of burnout (emotional exhaustion and depersonalization) as separate indicator variables and, alternatively, as continuous scales. We also included depression score as a continuous variable. For the long-term analyses, these variables were entered as changes in emotional exhaustion, depersonalization, and depression scores.

The original CHANGES cohort was selected using a stratified sampling design in which all medical students at geographically diverse set of schools were invited to participate; however, we have not incorporated sampling or nonresponse rates into the current analysis. Not doing so implies that estimates and inferences are valid only for the survey respondents and not for the general population of resident physicians; however, we selected this approach because respondents are now grouped by residency program rather than by school, with medical school explaining almost none of the variance in any of the key dependent or independent variables in the present study. Two-sided P < .05 were interpreted as statistically significant. All analyses were performed with Stata, version 15.1 (StataCorp LLC).

Results
Cross-sectional Cohort

The demographic characteristics and specialty training distribution of the 3392 nonblack second-year resident physicians (Table 1) were generally similar to those of all resident physicians in the United States.62,63 Of the 3392 participants, 1693 (49.9%) were male, 1964 (57.9%) were younger than 30 years, and 2362 (69.6%) self-identified as belonging to the white race.

In this cross-sectional cohort of 3380 resident physicians, 1203 (35.6%) had high emotional exhaustion, with a mean (SD) score of 3.0 (1.6) on the single item for emotional exhaustion; 1179 (34.9%) had high depersonalization, with a mean (SD) score of 2.9 (1.7) on the single item for depersonalization; and 1529 (45.2%) had burnout. Depressive symptoms were present in 1394 of 3377 resident physicians (41.3%), and the mean (SD) score on the PROMIS scale was 7.1 (3.3). Mean (SD) FT score toward black people was 77.9 (21.0), mean (SD) FT score toward white people was 81.1 (20.1), and mean (SD) racial IAT score was 0.4 (0.4), all of which indicate a preference for white people over black people. Histograms of R2 Questionnaire FT scores toward black people and white people, R2 Questionnaire IAT scores, and a scatterplot of FT score toward black people compared with FT score toward white people can be found in eFigure 1, eFigure 2, eFigure 3, and eFigure 4, respectively, in the Supplement.

Higher emotional exhaustion and depersonalization scores were associated with more unfavorable attitudes toward black people (as indicated by lower FT scores) (Figure 2). Resident physicians with high emotional exhaustion had lower mean (SD) FT scores toward black people compared with resident physicians without high emotional exhaustion (75.9 [21.9] vs 78.9 [20.4]; difference, –3.0; 95% CI, –4.5 to –1.5; P < .001). Similarly, resident physicians with high depersonalization had lower mean (SD) FT scores toward black people compared with resident physicians without high depersonalization (74.8 [22.3] vs 79.5 [20.1]; difference, –4.7; 95% CI, –6.2 to –3.2; P < .001). Overall, resident physicians who had at least 1 symptom of burnout had lower mean (SD) FT scores toward black people compared with those without symptoms of burnout (75.9 [21.9] vs 79.5 [20.1]; difference, –3.6; 95% CI, –5.0 to –2.2; P < .001). Resident physicians with depressive symptoms also had lower mean (SD) FT scores toward black people (74.9 [22.2] vs 80.0 [19.8]; difference, –5.0; 95% CI, –6.5 to –3.6; P < .001). On multivariable analysis, burnout (difference in FT score, –2.40; 95% CI, –3.42 to –1.37; P < .001; Table 2) and, in particular, depersonalization (for each 1-point increase, the difference in FT score decreased, –0.83; 95% CI, –1.22 to –0.45; P < .001; eTable 1 in the Supplement) were independently associated with lower FT score toward black people.

Implicit bias toward black people was also greater among resident physicians with high depersonalization (racial IAT mean [SD] scores, 0.48 [0.41] vs 0.42 [0.42]; difference, 0.05; 95% CI, 0.02-0.09; P < .001) and overall burnout (mean [SD], 0.47 [0.42] vs 0.42 [0.42]; difference, 0.05; 95% CI, 0.02-0.07; P = .002). No statistically significant difference in implicit bias toward black people was found among resident physicians with or without high emotional exhaustion (mean [SD], 0.46 [0.42] vs 0.43 [0.42]; difference, 0.03; 95% CI, 0.00-0.06; P = .07) and with or without depressive symptoms (mean [SD], 0.44 [0.42] vs 0.44 [0.42]; difference, 0.00; 95% CI, –0.03 to 0.03; P = .82). On multivariable analysis, burnout (difference in IAT score, 0.05; 95% CI, 0.02-0.08; P = .002; Table 2) and, in particular, depersonalization (for each 1-point increase, the difference in IAT score increased, 0.02; 95% CI, 0.01-0.03; P < .001; eTable 1 in the Supplement) were independently associated with implicit bias toward black people.

Long-term Cohort

In 2017, a total of 3058 resident physicians completed the R3 Questionnaire, among whom 2888 (94.4%) had completed the R2 Questionnaire. The 144 respondents who indicated their race was black were excluded, resulting in 2744 resident physicians comprising the long-term cohort. The demographic characteristics of the resident physicians providing long-term data were similar to those who provided only R2 Questionnaire data on age, race, ethnicity, parental status, and relationship status; however, women were more likely to provide long-term data (279 women respondents [43.1%] for R2 Questionnaire only vs 1404 women respondents [51.2%] for R2 and R3 Questionnaires; P = .001). The cohort had differences in specialty, as resident physicians who indicated on the R2 Questionnaire that they were training in pathology and radiology were not invited to complete the R3 Questionnaire (eTable 2 in the Supplement).

Among the 2733 resident physicians in this long-term cohort, 884 (33.3%) had symptoms of chronic burnout, 381 (13.9%) had recovered from symptoms of burnout, 346 (10.2%) had new symptoms of burnout, and 1122 (41.1%) never had symptoms of burnout. Mean (SD) FT scores toward black people increased from the R2 Questionnaire time point to the R3 Questionnaire time point (77.9 [21.0] vs 80.9 [20.0]; mean difference, 2.9 [19.2]; P < .001). Mean (SD) FT scores toward white people also increased from the R2 Questionnaire to the R3 Questionnaire time point (81.0 [20.1] vs 82.3 [19.4]; mean difference, 1.1 [19.4]; P < .01).

Mean (SD) FT scores toward black people at each time point for resident physicians who had symptoms of chronic burnout, had recovered from burnout, had new burnout, and never had burnout are shown in eFigure 5 in the Supplement. Resident physicians who never had burnout had higher mean FT scores toward black people at both R2 and R3 Questionnaire time points (80.2 and 82.7) compared with those who recovered from burnout (76.2 and 81.2), had new burnout (78.0 and 79.7), and had chronic burnout (76.0 and 78.9). Resident physicians who recovered from burnout had the highest gain in mean FT scores toward black people over the course of 1 year (4.8) compared with those who never had burnout (2.8), had new burnout (1.6), and had chronic burnout (2.9).

On multivariable analysis, the R3 Questionnaire FT score toward black people did not differ statistically across different burnout change patterns from the 2 time points, although the most favorable point-estimate implication was seen among resident physicians experiencing recovery from burnout (referent never had burnout; δ FT scores: recovered from burnout, 0.82 [95% CI, –1.17 to 2.80]; new burnout, –1.78 [95% CI, –3.83 to 0.28]; chronic burnout, –1.10 [95% CI, –2.69 to 0.48]; overall P = .10). However, a dose-response association was found between change in depersonalization from R2 to R3 Questionnaires and R3 Questionnaire explicit bias (for each 1-point increase in depersonalization, the difference in R3 FT score was –0.73; 95% CI, –1.23 to –0.23; P = .004) and change in explicit bias (Table 3).

Discussion

In this large national study of US resident physicians, reported symptoms of burnout were associated with greater explicit and implicit biases toward black people. Generally, feelings toward black people became more favorable from the second to the third year of residency. Worsening of depersonalization symptoms was statistically significantly associated with explicit bias toward black people.

These findings suggest that resident physicians’ feelings toward black people can become more favorable over the course of 1 year. This improvement may be associated with positive experiences with black people (colleagues, coworkers, and patients), graduate medical education efforts to reduce racial biases, changing signals of contextual antibias norms, and other factors.47,58,64 Whether the association between burnout and bias is causal is unclear, and both burnout and bias may be associated with the factors just described. However, these findings are consistent with those of other studies, which reported that positive emotions are associated with decreases in bias,65,66 suggesting that successful efforts to reduce symptoms of burnout among resident physicians may be useful in reducing health care inequalities.

We also found that implicit racial bias was higher among resident physicians with symptoms of burnout. According to previous studies, physicians with higher implicit bias toward black people demonstrate fewer patient-centered behaviors during clinical interactions with black patients; in turn, their black patients have greater distrust, have lower level of adherence to treatment recommendations, and are less likely to follow up.6,13,18,67-76 These data suggest symptoms of burnout may be associated with negative outcomes for black patients.

If the association between burnout and bias toward black people is present among physicians in practice or after residency, it may be a factor in the explicit use of race in medical decision-making. As the prevalence of burnout symptoms among practicing physicians exceeds 40%,77 the implications for the quality of care provided to black people, as well as to other disadvantaged groups, could be substantial. Further study is warranted to establish whether the association between burnout and racial bias persists among physicians in practice.

The findings of this study add to previous studies showing burnout as a threat to safe, high-quality care.78 Unfortunately, because little is known about the most effective strategies to reduce the prevalence of burnout,79,80 additional research into mitigating work-related drivers of burnout is needed.

Limitations

This study has several limitations. First, we assessed only a limited number of factors that may be associated with explicit and implicit biases. Second, we relied on measures of bias rather than on observed behaviors. Third, although the findings suggest an association between symptoms of burnout and bias toward black people, the magnitude of the observed associations was small to medium,81 and we could not ascertain if these associations were causal. In addition, minimal clinically important differences for FT and IAT scores have not been established in this population. Fourth, the generalizability of the results is unknown. However, the participation and questionnaire response rates were high, and resident physicians attended a wide range of medical schools and training programs. The demographic characteristics and specialty training distribution of the cohort were generally similar to all US medical residents. Fifth, differences in sex and specialty distributions were observed between respondents to only R2 Questionnaire and respondents to both R2 and R3 Questionnaires (eTable 2 in the Supplement). These differences may be explained by the men being less likely than the women in this study to complete questionnaires and the R3 Questionnaire not being sent to resident physicians who were training in radiology or pathology (excluded because they had less direct patient interaction).

Conclusions

Among US resident physicians, explicit bias and implicit bias were associated with symptoms of burnout. Given the high prevalence of burnout among physicians and the negative implications of bias for medical care, symptoms of burnout may be factors in disparities in care. The implications for the quality of care provided to black people and other disadvantaged groups could be substantial.

Back to top
Article Information

Accepted for Publication: May 25, 2019.

Published: July 26, 2019. doi:10.1001/jamanetworkopen.2019.7457

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2019 Dyrbye L et al. JAMA Network Open.

Corresponding Author: Liselotte Dyrbye, MD, MHPE, Division of Community Internal Medicine, Department of Medicine, Mayo Clinic, 200 First St SW, Rochester, MN 55905 (dyrbye.liselotte@mayo.edu).

Author Contributions: Dr Herrin had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Dyrbye, Dovidio, Hardeman, Burke, Phelan, van Ryn.

Acquisition, analysis, or interpretation of data: Dyrbye, Herrin, West, Wittlin, Dovidio, Hardeman, Burke, Phelan, Onyeador, Cunningham.

Drafting of the manuscript: Dyrbye, Dovidio, Hardeman.

Critical revision of the manuscript for important intellectual content: Dyrbye, Herrin, West, Wittlin, Hardeman, Burke, Phelan, Onyeador, Cunningham, van Ryn.

Statistical analysis: Herrin, West.

Obtained funding: Dyrbye, Dovidio, van Ryn.

Administrative, technical, or material support: Dyrbye, Phelan.

Supervision: Dyrbye.

Conflict of Interest Disclosures: Dr Dyrbye reported royalties for the Well-Being Index licensed by the Mayo Clinic to CWS Inc outside of the submitted work. Dr van Ryn reports being the founder and president of Diversity Science, a public benefit corporation (diversityscience.org) and Principal Investigator of the study that is the source of data for this manuscript.No other disclosures were reported.

Funding/Support: This study was supported in part by grant R01HL085631 from the National Heart, Lung and Blood Institute of the National Institutes of Health, and by the Mayo Clinic Department of Medicine Program on Physician Well-Being. Dr Hardeman was supported by the National Heart, Lung and Blood Institute through a Research Supplement to Promote Diversity in Health-Related Research (3R01HL085631-S2).

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

References
1.
Al-Khatib  SM, Fonarow  GC, Hayes  DL,  et al.  Performance measures to promote quality improvement in sudden cardiac arrest prevention and treatment.  Am Heart J. 2013;165(6):862-868. doi:10.1016/j.ahj.2013.02.010PubMedGoogle ScholarCrossref
2.
Kong  MH, Peterson  ED, Fonarow  GC,  et al.  Addressing disparities in sudden cardiac arrest care and the underutilization of effective therapies.  Am Heart J. 2010;160(4):605-618. doi:10.1016/j.ahj.2010.07.011PubMedGoogle ScholarCrossref
3.
Peterson  E, Yancy  CW.  Eliminating racial and ethnic disparities in cardiac care.  N Engl J Med. 2009;360(12):1172-1174. doi:10.1056/NEJMp0810121PubMedGoogle ScholarCrossref
4.
Yancy  CW.  Disparate care for acute myocardial infarction: moving beyond description and targeting interventions.  Circulation. 2014;130(8):632-633. doi:10.1161/CIRCULATIONAHA.114.011482PubMedGoogle ScholarCrossref
5.
Yancy  CW, Wang  TY, Ventura  HO,  et al; credo Advisory Group.  The coalition to reduce racial and ethnic disparities in cardiovascular disease outcomes (credo): why credo matters to cardiologists.  J Am Coll Cardiol. 2011;57(3):245-252. doi:10.1016/j.jacc.2010.09.027PubMedGoogle ScholarCrossref
6.
Dovidio  JF, Fiske  ST.  Under the radar: how unexamined biases in decision-making processes in clinical interactions can contribute to health care disparities.  Am J Public Health. 2012;102(5):945-952. doi:10.2105/AJPH.2011.300601PubMedGoogle ScholarCrossref
7.
Esnaola  NF, Ford  ME.  Racial differences and disparities in cancer care and outcomes: where’s the rub?  Surg Oncol Clin N Am. 2012;21(3):417-437, viii. doi:10.1016/j.soc.2012.03.012PubMedGoogle ScholarCrossref
8.
Mehta  JL, Bursac  Z, Mehta  P,  et al.  Racial disparities in prescriptions for cardioprotective drugs and cardiac outcomes in Veterans Affairs Hospitals.  Am J Cardiol. 2010;105(7):1019-1023. doi:10.1016/j.amjcard.2009.11.031PubMedGoogle ScholarCrossref
9.
Moy  E, Freeman  W.  Federal investments to eliminate racial/ethnic health-care disparities.  Public Health Rep. 2014;129(suppl 2):62-70. doi:10.1177/00333549141291S212PubMedGoogle ScholarCrossref
10.
Saha  S, Korthuis  PT, Cohn  JA, Sharp  VL, Moore  RD, Beach  MC.  Primary care provider cultural competence and racial disparities in HIV care and outcomes.  J Gen Intern Med. 2013;28(5):622-629. doi:10.1007/s11606-012-2298-8PubMedGoogle ScholarCrossref
11.
van Ryn  M, Burgess  DJ, Dovidio  JF,  et al.  The impact of racism on clinician cognition, behavior, and clinical decision-making.  Du Bois Rev. 2011;8(1):199-218. doi:10.1017/S1742058X11000191PubMedGoogle ScholarCrossref
12.
Epstein  RA.  Disparities and discrimination in health care coverage: a critique of the Institute of Medicine study.  Perspect Biol Med. 2005;48(1)(suppl):S26-S41. doi:10.1353/pbm.2005.0023PubMedGoogle ScholarCrossref
13.
Penner  LA, Dovidio  JF, West  TV,  et al.  Aversive racism and medical interactions with black patients: a field study.  J Exp Soc Psychol. 2010;46(2):436-440. doi:10.1016/j.jesp.2009.11.004PubMedGoogle ScholarCrossref
14.
Paradies  Y, Truong  M, Priest  N.  A systematic review of the extent and measurement of healthcare provider racism.  J Gen Intern Med. 2014;29(2):364-387. doi:10.1007/s11606-013-2583-1PubMedGoogle ScholarCrossref
15.
van Ryn  M, Fu  SS.  Paved with good intentions: do public health and human service providers contribute to racial/ethnic disparities in health?  Am J Public Health. 2003;93(2):248-255. doi:10.2105/AJPH.93.2.248PubMedGoogle ScholarCrossref
16.
Dovidio  JF, ten Vergert  M, Stewart  TL,  et al.  Perspective and prejudice: antecedents and mediating mechanisms.  Pers Soc Psychol Bull. 2004;30(12):1537-1549. doi:10.1177/0146167204271177PubMedGoogle ScholarCrossref
17.
Penner  LA, Gaertner  S, Dovidio  JF,  et al.  A social psychological approach to improving the outcomes of racially discordant medical interactions.  J Gen Intern Med. 2013;28(9):1143-1149. doi:10.1007/s11606-013-2339-yPubMedGoogle ScholarCrossref
18.
Blair  IV, Steiner  JF, Fairclough  DL,  et al.  Clinicians’ implicit ethnic/racial bias and perceptions of care among Black and Latino patients.  Ann Fam Med. 2013;11(1):43-52. doi:10.1370/afm.1442PubMedGoogle ScholarCrossref
19.
Green  AR, Carney  DR, Pallin  DJ,  et al.  Implicit bias among physicians and its prediction of thrombolysis decisions for black and white patients.  J Gen Intern Med. 2007;22(9):1231-1238. doi:10.1007/s11606-007-0258-5PubMedGoogle ScholarCrossref
20.
Sabin  J, Nosek  BA, Greenwald  A, Rivara  FP.  Physicians’ implicit and explicit attitudes about race by MD race, ethnicity, and gender.  J Health Care Poor Underserved. 2009;20(3):896-913. doi:10.1353/hpu.0.0185PubMedGoogle ScholarCrossref
21.
Sabin  JA, Rivara  FP, Greenwald  AG.  Physician implicit attitudes and stereotypes about race and quality of medical care.  Med Care. 2008;46(7):678-685. doi:10.1097/MLR.0b013e3181653d58PubMedGoogle ScholarCrossref
22.
Sabin  JA, Greenwald  AG.  The influence of implicit bias on treatment recommendations for 4 common pediatric conditions: pain, urinary tract infection, attention deficit hyperactivity disorder, and asthma.  Am J Public Health. 2012;102(5):988-995. doi:10.2105/AJPH.2011.300621PubMedGoogle ScholarCrossref
23.
Dehon  E, Weiss  N, Jones  J, Faulconer  W, Hinton  E, Sterling  S.  A systematic review of the impact of physician implicit racial bias on clinical decision-making.  Acad Emerg Med. 2017;24(8):895-904. doi:10.1111/acem.13214PubMedGoogle ScholarCrossref
24.
Finucane  TE, Carrese  JA.  Racial bias in presentation of cases.  J Gen Intern Med. 1990;5(2):120-121. doi:10.1007/BF02600511PubMedGoogle ScholarCrossref
25.
Haider  AH, Sexton  J, Sriram  N,  et al.  Association of unconscious race and social class bias with vignette-based clinical assessments by medical students.  JAMA. 2011;306(9):942-951. doi:10.1001/jama.2011.1248PubMedGoogle ScholarCrossref
26.
Smith  WR, Betancourt  JR, Wynia  MK,  et al.  Recommendations for teaching about racial and ethnic disparities in health and health care.  Ann Intern Med. 2007;147(9):654-665. doi:10.7326/0003-4819-147-9-200711060-00010PubMedGoogle ScholarCrossref
27.
Stone  J, Moskowitz  GB.  Non-conscious bias in medical decision-making: what can be done to reduce it?  Med Educ. 2011;45(8):768-776. doi:10.1111/j.1365-2923.2011.04026.xPubMedGoogle ScholarCrossref
28.
Byrne  A, Tanesini  A.  Instilling new habits: addressing implicit bias in healthcare professionals.  Adv Health Sci Educ Theory Pract. 2015;20(5):1255-1262. doi:10.1007/s10459-015-9600-6PubMedGoogle ScholarCrossref
29.
Dyrbye  LN, Burke  SE, Hardeman  RR,  et al.  Association of clinical specialty with symptoms of burnout and career choice regret among us resident physicians [ retracted in: JAMA. 2019;321(12):1220-1221].  JAMA. 2018;320(11):1114-1130. doi:10.1001/jama.2018.12615PubMedGoogle ScholarCrossref
30.
West  CP, Shanafelt  TD, Kolars  JC.  Quality of life, burnout, educational debt, and medical knowledge among internal medicine residents.  JAMA. 2011;306(9):952-960. doi:10.1001/jama.2011.1247PubMedGoogle ScholarCrossref
31.
Thomas  NK.  Resident burnout.  JAMA. 2004;292(23):2880-2889. doi:10.1001/jama.292.23.2880PubMedGoogle ScholarCrossref
32.
Mata  DA, Ramos  MA, Bansal  N,  et al.  Prevalence of depression and depressive symptoms among resident physicians: a systematic review and meta-analysis.  JAMA. 2015;314(22):2373-2383. doi:10.1001/jama.2015.15845PubMedGoogle ScholarCrossref
33.
Dyrbye  L, Shanafelt  T.  A narrative review on burnout experienced by medical students and residents.  Med Educ. 2016;50(1):132-149. doi:10.1111/medu.12927PubMedGoogle ScholarCrossref
34.
Maslach  C, Jackson  SE, Leiter  MP.  Maslach Burnout Inventory. 4th ed. Menlo Park, CA: Mind Garden, Inc; 2016.
35.
Cunningham  BA, Bonham  VL, Sellers  SL, Yeh  HC, Cooper  LA.  Physicians’ anxiety due to uncertainty and use of race in medical decision-making.  Med Care. 2014;52(8):728-733. doi:10.1097/MLR.0000000000000157PubMedGoogle ScholarCrossref
36.
McConnell  MM, Eva  KW.  The role of emotion in the learning and transfer of clinical skills and knowledge.  Acad Med. 2012;87(10):1316-1322. doi:10.1097/ACM.0b013e3182675af2PubMedGoogle ScholarCrossref
37.
Kuhlmann  S, Piel  M, Wolf  OT.  Impaired memory retrieval after psychosocial stress in healthy young men.  J Neurosci. 2005;25(11):2977-2982. doi:10.1523/JNEUROSCI.5139-04.2005PubMedGoogle ScholarCrossref
38.
Smallwood  J, Fitzgerald  A, Miles  LK, Phillips  LH.  Shifting moods, wandering minds: negative moods lead the mind to wander.  Emotion. 2009;9(2):271-276. doi:10.1037/a0014855PubMedGoogle ScholarCrossref
39.
Dasgupta  N, Desteno  D, Williams  LA, Hunsinger  M.  Fanning the flames of prejudice: the influence of specific incidental emotions on implicit prejudice.  Emotion. 2009;9(4):585-591. doi:10.1037/a0015961PubMedGoogle ScholarCrossref
40.
von Elm  E, Altman  DG, Egger  M, Pocock  SJ, Gøtzsche  PC, Vandenbroucke  JP; STROBE Initiative.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.  Ann Intern Med. 2007;147(8):573-577. doi:10.7326/0003-4819-147-8-200710160-00010PubMedGoogle ScholarCrossref
41.
American Association for Public Opinion Research.  Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 9th ed. https://www.aapor.org/AAPOR_Main/media/publications/Standard-Definitions20169theditionfinal.pdf. Revised 2016. Accessed June 20, 2019.
42.
Hardeman  RR, Burgess  D, Phelan  S, Yeazel  M, Nelson  D, van Ryn  M.  Medical student socio-demographic characteristics and attitudes toward patient centered care: do race, socioeconomic status and gender matter? A report from the Medical Student CHANGES study.  Patient Educ Couns. 2015;98(3):350-355. doi:10.1016/j.pec.2014.11.013PubMedGoogle ScholarCrossref
43.
Hardeman  RR, Przedworski  JM, Burke  SE,  et al.  Mental well-being in first year medical students: a comparison by race and gender: a report from the medical student CHANGE study.  J Racial Ethn Health Disparities. 2015;2(3):403-413. doi:10.1007/s40615-015-0087-xPubMedGoogle ScholarCrossref
44.
Phelan  SM, Burgess  DJ, Puhl  R,  et al.  The adverse effect of weight stigma on the well-being of medical students with overweight or obesity: findings from a national survey.  J Gen Intern Med. 2015;30(9):1251-1258. doi:10.1007/s11606-015-3266-xPubMedGoogle ScholarCrossref
45.
Phelan  SM, Dovidio  JF, Puhl  RM,  et al.  Implicit and explicit weight bias in a national sample of 4,732 medical students: the medical student CHANGES study.  Obesity (Silver Spring). 2014;22(4):1201-1208. doi:10.1002/oby.20687PubMedGoogle ScholarCrossref
46.
Phelan  SM, Puhl  RM, Burke  SE,  et al.  The mixed impact of medical school on medical students’ implicit and explicit weight bias.  Med Educ. 2015;49(10):983-992. doi:10.1111/medu.12770PubMedGoogle ScholarCrossref
47.
van Ryn  M, Hardeman  R, Phelan  SM,  et al.  Medical school experiences associated with change in implicit racial bias among 3547 students: a medical student CHANGES study report.  J Gen Intern Med. 2015;30(12):1748-1756. doi:10.1007/s11606-015-3447-7PubMedGoogle ScholarCrossref
48.
van Ryn  M, Hardeman  RR, Phelan  SM,  et al.  Psychosocial predictors of attitudes toward physician empathy in clinical encounters among 4732 1st year medical students: a report from the CHANGES study.  Patient Educ Couns. 2014;96(3):367-375. doi:10.1016/j.pec.2014.06.009PubMedGoogle ScholarCrossref
49.
Laveist  TA, Nuru-Jeter  A.  Is doctor-patient race concordance associated with greater satisfaction with care?  J Health Soc Behav. 2002;43(3):296-306. doi:10.2307/3090205PubMedGoogle ScholarCrossref
50.
Thornton  RLJ, Powe  NR, Roter  D, Cooper  LA.  Patient-physician social concordance, medical visit communication and patients’ perceptions of health care quality.  Patient Educ Couns. 2011;85(3):e201-e208. doi:10.1016/j.pec.2011.07.015PubMedGoogle ScholarCrossref
51.
Shanafelt  TD, Hasan  O, Dyrbye  LN,  et al.  Changes in burnout and satisfaction with work-life balance in physicians and the general US working population between 2011 and 2014.  [Erratum appears in Mayo Clin Proc. 91(2):276].  Mayo Clin Proc. 2015;90(12):1600-1613. doi:10.1016/j.mayocp.2015.08.023PubMedGoogle ScholarCrossref
52.
Shanafelt  TD, Boone  S, Tan  L,  et al.  Burnout and satisfaction with work-life balance among US physicians relative to the general US population.  Arch Intern Med. 2012;172(18):1377-1385. doi:10.1001/archinternmed.2012.3199PubMedGoogle ScholarCrossref
53.
West  CP, Dyrbye  LN, Satele  DV, Sloan  JA, Shanafelt  TD.  Concurrent validity of single-item measures of emotional exhaustion and depersonalization in burnout assessment.  J Gen Intern Med. 2012;27(11):1445-1452. doi:10.1007/s11606-012-2015-7PubMedGoogle ScholarCrossref
54.
West  CP, Dyrbye  LN, Sloan  JA, Shanafelt  TD.  Single item measures of emotional exhaustion and depersonalization are useful for assessing burnout in medical professionals.  J Gen Intern Med. 2009;24(12):1318-1321. doi:10.1007/s11606-009-1129-zPubMedGoogle ScholarCrossref
55.
Pilkonis  PA, Choi  SW, Reise  SP, Stover  AM, Riley  WT, Cella  D; PROMIS Cooperative Group.  Item banks for measuring emotional distress from the Patient-Reported Outcomes Measurement Information System (PROMIS®): depression, anxiety, and anger.  Assessment. 2011;18(3):263-283. doi:10.1177/1073191111411667PubMedGoogle ScholarCrossref
56.
Kroenke  K, Yu  Z, Wu  J, Kean  J, Monahan  PO.  Operating characteristics of PROMIS four-item depression and anxiety scales in primary care patients with chronic pain.  Pain Med. 2014;15(11):1892-1901. doi:10.1111/pme.12537PubMedGoogle ScholarCrossref
57.
Alwin  D.  Feeling thermometers versus 7-point scales: which are better.  Sociol Methods Res. 1997;25:318-340. doi:10.1177/0049124197025003003Google ScholarCrossref
58.
Burke  SE, Dovidio  JF, Perry  SP,  et al.  Informal training experiences and explicit bias against African Americans among medical students.  Soc Psychol Q. 2017;80:65-84. doi:10.1177/0190272516668166Google ScholarCrossref
59.
Greenwald  AG, McGhee  DE, Schwartz  JLK.  Measuring individual differences in implicit cognition: the implicit association test.  J Pers Soc Psychol. 1998;74(6):1464-1480. doi:10.1037/0022-3514.74.6.1464PubMedGoogle ScholarCrossref
60.
Greenwald  AG, Nosek  BA, Banaji  MR.  Understanding and using the implicit association test: I. An improved scoring algorithm.  J Pers Soc Psychol. 2003;85(2):197-216. doi:10.1037/0022-3514.85.2.197PubMedGoogle ScholarCrossref
61.
Chapman  EN, Kaatz  A, Carnes  M.  Physicians and implicit bias: how doctors may unwittingly perpetuate health care disparities.  J Gen Intern Med. 2013;28(11):1504-1510. doi:10.1007/s11606-013-2441-1PubMedGoogle ScholarCrossref
62.
Accreditation Council for Graduate Medical Education. Data Resource Book. 2010-2011. https://www.acgme.org/About-Us/Publications-and-Resources/Graduate-Medical-Education-Data-Resource-Book. Accessed May 21, 2018
63.
Association of American Medical Colleges. Medical student education: debt, costs, and loan repayment fact card. https://store.aamc.org/downloadable/download/sample/sample_id/240/. Accessed May 21, 2018
64.
Pettigrew  TF, Tropp  LR.  A meta-analytic test of intergroup contact theory.  J Pers Soc Psychol. 2006;90(5):751-783. doi:10.1037/0022-3514.90.5.751PubMedGoogle ScholarCrossref
65.
Johnson  KJ, Fredrickson  BL.  “We all look the same to me”: positive emotions eliminate the own-race in face recognition.  Psychol Sci. 2005;16(11):875-881. doi:10.1111/j.1467-9280.2005.01631.xPubMedGoogle ScholarCrossref
66.
Kang  Y, Gray  JR, Dovidio  JF.  The nondiscriminating heart: lovingkindness meditation training decreases implicit intergroup bias.  J Exp Psychol Gen. 2014;143(3):1306-1313. doi:10.1037/a0034150PubMedGoogle ScholarCrossref
67.
Hagiwara  N, Penner  LA, Gonzalez  R,  et al.  Racial attitudes, physician-patient talk time ratio, and adherence in racially discordant medical interactions.  Soc Sci Med. 2013;87:123-131. doi:10.1016/j.socscimed.2013.03.016PubMedGoogle ScholarCrossref
68.
Eggly  S, Barton  E, Winckles  A, Penner  LA, Albrecht  TL.  A disparity of words: racial differences in oncologist-patient communication about clinical trials.  Health Expect. 2015;18(5):1316-1326. doi:10.1111/hex.12108PubMedGoogle ScholarCrossref
69.
Penner  LA, Dovidio  JF, Edmondson  D,  et al.  The experience of discrimination and black-white health disparities in medical care.  J Black Psychol. 2009;35(2). doi:10.1177/0095798409333585PubMedGoogle Scholar
70.
Cooper  LA, Roter  DL, Carson  KA,  et al.  The associations of clinicians’ implicit attitudes about race with medical visit communication and patient ratings of interpersonal care.  Am J Public Health. 2012;102(5):979-987. doi:10.2105/AJPH.2011.300558PubMedGoogle ScholarCrossref
71.
Dovidio  JF, Penner  LA, Albrecht  TL, Norton  WE, Gaertner  SL, Shelton  JN.  Disparities and distrust: the implications of psychological processes for understanding racial disparities in health and health care.  Soc Sci Med. 2008;67(3):478-486. doi:10.1016/j.socscimed.2008.03.019PubMedGoogle ScholarCrossref
72.
Cuffee  YL, Hargraves  JL, Rosal  M,  et al.  Reported racial discrimination, trust in physicians, and medication adherence among inner-city African Americans with hypertension.  Am J Public Health. 2013;103(11):e55-e62. doi:10.2105/AJPH.2013.301554PubMedGoogle ScholarCrossref
73.
Kronish  IM, Diefenbach  MA, Edmondson  DE, Phillips  LA, Fei  K, Horowitz  CR.  Key barriers to medication adherence in survivors of strokes and transient ischemic attacks.  J Gen Intern Med. 2013;28(5):675-682. doi:10.1007/s11606-012-2308-xPubMedGoogle ScholarCrossref
74.
Martin  KD, Roter  DL, Beach  MC, Carson  KA, Cooper  LA.  Physician communication behaviors and trust among black and white patients with hypertension.  Med Care. 2013;51(2):151-157. doi:10.1097/MLR.0b013e31827632a2PubMedGoogle ScholarCrossref
75.
Amy  NK, Aalborg  A, Lyons  P, Keranen  L.  Barriers to routine gynecological cancer screening for White and African-American obese women.  Int J Obes (Lond). 2006;30(1):147-155. doi:10.1038/sj.ijo.0803105PubMedGoogle ScholarCrossref
76.
Penner  LA, Phelan  SM, Earnshaw  V, Albrecht  TL, Dovidio  JF. Patient stigma, medical interactions, and health care disparities: a selective review. In: Major B, Dovidio JF, Link BG, eds. The Oxford Handbook of Stigma, Discrimination, and Health. New York, NY: Oxford University Press; 2017:183-201.
77.
Shanafelt  T, West  CP, Sinsky  C,  et al.  Changes in burnout and satisfaction with work-life integration in physicians and the general US working population between 2011-2017.  Mayo Clin Proc. 2019;pii:S0025-6196(18)30938-8.Google Scholar
78.
Dyrbye  LN, Shanafelt  TD, Sinsky  CA,  et al.  Burnout Among Health Care Professionals: A Call to Explore and Address This Underrecognized Threat to Safe, High-Quality Care. Washington, DC: National Academy of Medicine; 2017.
79.
West  CP, Dyrbye  LN, Erwin  PJ, Shanafelt  TD.  Interventions to prevent and reduce physician burnout: a systematic review and meta-analysis.  Lancet. 2016;388(10057):2272-2281. doi:10.1016/S0140-6736(16)31279-XPubMedGoogle ScholarCrossref
80.
Panagioti  M, Panagopoulou  E, Bower  P,  et al.  Controlled interventions to reduce burnout in physicians: a systematic review and meta-analysis.  JAMA Intern Med. 2017;177(2):195-205. doi:10.1001/jamainternmed.2016.7674PubMedGoogle ScholarCrossref
81.
Cohen  J.  Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Abingdon-on-Thames, UK: Routledge; 1988.
×