Clinician Conceptualization of the Benefits of Treatments for Individual Patients | Cardiology | JAMA Network Open | JAMA Network
[Skip to Navigation]
Sign In
Figure 1.  Clinician Estimates of Likelihood of Disease Outcomes and Benefits of Treatment
Clinician Estimates of Likelihood of Disease Outcomes and Benefits of Treatment

Gray shaded areas indicate the frequency distribution of responses; the dotted vertical line identifies the median answer.

Figure 2.  Association Between Clinician Estimate of Benefit and Use of Treatment in Real Life Patients
Association Between Clinician Estimate of Benefit and Use of Treatment in Real Life Patients

The boxes represent the middle 50% of data, with the lines representing the median. Diamonds indicate means, and dots indicate values outside the interquartile range.

Table 1.  Recruitment of Participants by Study Site and Reasons for Lack of Participation
Recruitment of Participants by Study Site and Reasons for Lack of Participation
Table 2.  Demographic Characteristics and Other Practice Factors Among Enrolled Health Care Workers
Demographic Characteristics and Other Practice Factors Among Enrolled Health Care Workers
1.
US Food and Drug Administration. The FDA’s drug review process: ensuring drugs are safe and effective. November 24, 2017. Accessed September 21, 2020. https://www.fda.gov/drugs/drug-information-consumers/fdas-drug-review-process-ensuring-drugs-are-safe-and-effective
2.
Armstrong  KA, Metlay  JP.  Annals clinical decision making: translating population evidence to individual patients.   Ann Intern Med. 2020;172(9):610-616. doi:10.7326/M19-3496PubMedGoogle ScholarCrossref
3.
Elliott  MH, Skydel  JJ, Dhruva  SS, Ross  JS, Wallach  JD.  Characteristics and reporting of number needed to treat, number needed to harm, and absolute risk reduction in controlled clinical trials, 2001-2019.   JAMA Intern Med. 2021;181(2):282-284. doi:10.1001/jamainternmed.2020.4799PubMedGoogle ScholarCrossref
4.
Zipkin  DA, Umscheid  CA, Keating  NL,  et al.  Evidence-based risk communication: a systematic review.   Ann Intern Med. 2014;161(4):270-280. doi:10.7326/M14-0295PubMedGoogle ScholarCrossref
5.
Gigerenzer  G.  Why does framing influence judgment?   J Gen Intern Med. 2003;18(11):960-961. doi:10.1046/j.1525-1497.2003.30901.xPubMedGoogle ScholarCrossref
6.
Paling  J.  Strategies to help patients understand risks.   BMJ. 2003;327(7417):745-748. doi:10.1136/bmj.327.7417.745PubMedGoogle ScholarCrossref
7.
Barratt  A, Trevena  L, Davey  HM, McCaffery  K.  Use of decision aids to support informed choices about screening.   BMJ. 2004;329(7464):507-510. doi:10.1136/bmj.329.7464.507PubMedGoogle ScholarCrossref
8.
Nuovo  J, Melnikow  J, Chang  D.  Reporting number needed to treat and absolute risk reduction in randomized controlled trials.   JAMA. 2002;287(21):2813-2814. doi:10.1001/jama.287.21.2813PubMedGoogle ScholarCrossref
9.
Epstein  D; ProPublica. When evidence says no, but doctors say yes. The Atlantic. February 22, 2017. Accessed August 18, 2020. https://www.theatlantic.com/health/archive/2017/02/when-evidence-says-no-but-doctors-say-yes/517368/
10.
Armstrong  KA, Metlay  JP.  Annals clinical decision making: communicating risk and engaging patients in shared decision making.   Ann Intern Med. 2020;172(10):688-692. doi:10.7326/M19-3495PubMedGoogle ScholarCrossref
11.
Morgan  DJ, Scherer  LD, Korenstein  D.  Improving physician communication about treatment decisions: reconsideration of “risks vs benefits.”   JAMA. 2020;324(10):937-938. doi:10.1001/jama.2020.0354PubMedGoogle ScholarCrossref
12.
Thomas  K, Fink  S. FDA “grossly misrepresented” blood plasma data, scientists say. The New York Times. August 24, 2020. Accessed September 21, 2020. https://www.nytimes.com/2020/08/24/health/fda-blood-plasma.html
13.
Hoffmann  TC, Del Mar  C.  Clinicians’ expectations of the benefits and harms of treatments, screening, and tests: a systematic review.   JAMA Intern Med. 2017;177(3):407-419. doi:10.1001/jamainternmed.2016.8254PubMedGoogle ScholarCrossref
14.
Treadwell  JS, Wong  G, Milburn-Curtis  C, Feakins  B, Greenhalgh  T.  GPs’ understanding of the benefits and harms of treatments for long-term conditions: an online survey.   BJGP Open. 2020;4(1):bjgpopen20X101016. doi:10.3399/bjgpopen20X101016PubMedGoogle Scholar
15.
Morgan  DJ, Pineles  L, Owczarzak  J,  et al.  Accuracy of practitioner estimates of probability of diagnosis before and after testing.   JAMA Intern Med. 2021;181(6):747-755. doi:10.1001/jamainternmed.2021.0269PubMedGoogle ScholarCrossref
16.
Gigerenzer  G.  Reckoning With Risk: Learning to Live with Uncertainty. Penguin Books; 2003.
17.
Krouss  M, Croft  L, Morgan  DJ.  Physician understanding and ability to communicate harms and benefits of common medical treatments.   JAMA Intern Med. 2016;176(10):1565-1567. doi:10.1001/jamainternmed.2016.5027PubMedGoogle ScholarCrossref
18.
Fagerlin  A, Sepucha  KR, Couper  MP, Levin  CA, Singer  E, Zikmund-Fisher  BJ.  Patients’ knowledge about 9 common health conditions: the DECISIONS survey.   Med Decis Making. 2010;30(5)(suppl):35S-52S. doi:10.1177/0272989X10378700PubMedGoogle Scholar
19.
The American Association for Public Opinion Research.  Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 9th edition. AAPOR; 2016.
20.
Woloshin  S, Schwartz  LM.  Communicating data about the benefits and harms of treatment: a randomized trial.   Ann Intern Med. 2011;155(2):87-96. doi:10.7326/0003-4819-155-2-201107190-00004PubMedGoogle ScholarCrossref
21.
Barratt  A, Wyer  PC, Hatala  R,  et al; Evidence-Based Medicine Teaching Tips Working Group.  Tips for learners of evidence-based medicine: 1. relative risk reduction, absolute risk reduction and number needed to treat.   CMAJ. 2004;171(4):353-358. doi:10.1503/cmaj.1021197PubMedGoogle ScholarCrossref
22.
Guyatt  GH, Oxman  AD, Vist  GE,  et al; GRADE Working Group.  GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.   BMJ. 2008;336(7650):924-926. doi:10.1136/bmj.39489.470347.ADPubMedGoogle ScholarCrossref
23.
World Health Organization/Health Action International. Understanding and responding to pharmaceutical promotion–a practical guide. Accessed December 16, 2020. https://www.paho.org/hq/dmdocuments/2011/drug-promotion-manual-CAP-3-090610.pdf
24.
Lexchin  J.  How patient outcomes are reported in drug advertisements.   Can Fam Physician. 1999;45:1213-1216.PubMedGoogle Scholar
25.
Rodman  A,. Episode 50: I know nothing. Bedside Rounds. October 27, 2019. Accessed November 24, 2020. http://bedside-rounds.org/episode-50-i-know-nothing/
26.
Prasad  V, Ho  N.  Why do we continue to adopt medical practices based on pathophysiology alone when we should be insisting on clinical trials?   J Clin Epidemiol. 2014;67(4):361-363. doi:10.1016/j.jclinepi.2013.11.009PubMedGoogle ScholarCrossref
27.
Montgomery  K.  How Doctors Think: Clinical Judgment and the Practice of Medicine. Oxford University Press; 2005.
28.
Khan  KS, Coomarasamy  A.  A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine.   BMC Med Educ. 2006;6:59. doi:10.1186/1472-6920-6-59PubMedGoogle ScholarCrossref
29.
Tversky  A, Kahneman  D.  Judgment under uncertainty: heuristics and biases.   Science. 1974;185(4157):1124-1131. doi:10.1126/science.185.4157.1124PubMedGoogle ScholarCrossref
30.
Saposnik  G, Redelmeier  D, Ruff  CC, Tobler  PN.  Cognitive biases associated with medical decisions: a systematic review.   BMC Med Inform Decis Mak. 2016;16(1):138. doi:10.1186/s12911-016-0377-1PubMedGoogle ScholarCrossref
31.
Crowley  RS, Legowski  E, Medvedeva  O,  et al.  Automated detection of heuristics and biases among pathologists in a computer-based system.   Adv Health Sci Educ Theory Pract. 2013;18(3):343-363. doi:10.1007/s10459-012-9374-zPubMedGoogle ScholarCrossref
32.
Kusev  P, van Schaik  P, Ayton  P, Dent  J, Chater  N.  Exaggerated risk: prospect theory and probability weighting in risky choice.   J Exp Psychol Learn Mem Cogn. 2009;35(6):1487-1505. doi:10.1037/a0017039PubMedGoogle ScholarCrossref
33.
DeJong  J. CHA2DS2-VASc/HAS-BLED/EHRA atrial fibrillation risk score calculator. Accessed June 16, 2021. https://www.chadsvasc.org/
34.
Viswanathan  M, Reddy  S, Berkman  N,  et al.  Screening to prevent osteoporotic fractures: updated evidence report and systematic review for the US Preventive Services Task Force.   JAMA. 2018;319(24):2532-2551. doi:10.1001/jama.2018.6537PubMedGoogle ScholarCrossref
35.
Whelton  PK, Carey  RM, Aronow  WS,  et al.  2017 ACC/AHA/AAPA/ABC/ACPM/AGS/APhA/ASH/ASPC/NMA/PCNA guideline for the prevention, detection, evaluation, and management of high blood pressure in adults: a report of the American College of Cardiology/American Heart Association task force on clinical practice guidelines.   Hypertension. 2018;71(6):e13-e115. doi:10.1161/HYP.0000000000000065PubMedGoogle Scholar
36.
Whelton  PK, Williams  B.  The 2018 European Society of Cardiology/European Society of Hypertension and 2017 American College of Cardiology/American Heart Association blood pressure guidelines: more similar than different.   JAMA. 2018;320(17):1749-1750. doi:10.1001/jama.2018.16755PubMedGoogle ScholarCrossref
37.
Morgan  DJ, Dhruva  SS, Wright  SM, Korenstein  D.  2016 Update on medical overuse: a systematic review.   JAMA Intern Med. 2016;176(11):1687-1692. doi:10.1001/jamainternmed.2016.5381PubMedGoogle ScholarCrossref
38.
Wells’ Criteria for Pulmonary Embolism. MDCalc. Accessed April 27, 2020. https://www.mdcalc.com/wells-criteria-pulmonary-embolism
39.
January  CT, Wann  LS, Calkins  H,  et al.  2019 AHA/ACC/HRS focused update of the 2014 AHA/ACC/HRS guideline for the management of patients with atrial fibrillation: a report of the American College of Cardiology/American Heart Association task force on clinical practice guidelines and the Heart Rhythm Society in collaboration with the Society of Thoracic Surgeons.   Circulation. 2019;140(2):e125-e151. doi:10.1161/CIR.0000000000000665PubMedGoogle ScholarCrossref
40.
Sheridan  SL, Donahue  KE, Brenner  AT.  Beginning with high value care in mind: a scoping review and toolkit to support the content, delivery, measurement, and sustainment of high value care.   Patient Educ Couns. 2019;102(2):238-252. doi:10.1016/j.pec.2018.05.014PubMedGoogle ScholarCrossref
41.
Morgan  DJ, Leppin  AL, Smith  CD, Korenstein  D.  A practical framework for understanding and reducing medical overuse: conceptualizing overuse through the patient-clinician interaction.   J Hosp Med. 2017;12(5):346-351. doi:10.12788/jhm.2738PubMedGoogle ScholarCrossref
42.
Grundy  SM, Stone  NJ, Bailey  AL,  et al.  2018 AHA/ACC/AACVPR/AAPA/ABC/ACPM/ADA/AGS/APhA/ASPC/NLA/PCNA guideline on the management of blood cholesterol: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines.   J Am Coll Cardiol. 2019;73(24):e285-e350. doi:10.1016/j.jacc.2018.11.003PubMedGoogle ScholarCrossref
43.
Sheridan  SL, Pignone  MP, Lewis  CL.  A randomized comparison of patients’ understanding of number needed to treat and other common risk reduction formats.   J Gen Intern Med. 2003;18(11):884-892. doi:10.1046/j.1525-1497.2003.21102.xPubMedGoogle ScholarCrossref
44.
Back  AL, Fromme  EK, Meier  DE.  Training clinicians with communication skills needed to match medical treatments to patient values.   J Am Geriatr Soc. 2019;67(S2):S435-S441. doi:10.1111/jgs.15709PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Views 12,962
    Citations 0
    Original Investigation
    Statistics and Research Methods
    July 21, 2021

    Clinician Conceptualization of the Benefits of Treatments for Individual Patients

    Author Affiliations
    • 1Department of Epidemiology and Public Health, University of Maryland School of Medicine, Baltimore
    • 2VA Maryland Healthcare System, Baltimore
    • 3Department of Health, Behavior and Society, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland
    • 4Adult and Child Consortium of Health Outcomes Research and Delivery Science (ACCORDS), University of Colorado School of Medicine, Aurora
    • 5Division of Cardiology, University of Colorado School of Medicine, Aurora
    • 6Center of Innovation for Veteran-Centered and Value-Driven Care, VA Denver, Denver, Colorado
    • 7Division of General Internal Medicine and Geriatrics, Department of Medicine, Oregon Health and Science University, Portland
    • 8Department of Medicine, Dell Medical School, the University of Texas at Austin
    • 9South Texas Veterans Health Care System, San Antonio
    • 10Department of Medicine, University of Wisconsin School of Medicine and Public Health, Madison
    • 11Department of Medicine, Penn State College of Medicine, Hershey, Pennsylvania
    • 12Department of Public Health Sciences, Penn State College of Medicine, Hershey, Pennsylvania
    • 13Department of Medicine, University of Maryland School of Medicine, Baltimore
    • 14Genomic Medicine Institute, Geisinger, Danville, Pennsylvania
    • 15New York University Grossman School of Medicine, New York
    • 16Memorial Sloan Kettering Cancer Center, New York, New York
    JAMA Netw Open. 2021;4(7):e2119747. doi:10.1001/jamanetworkopen.2021.19747
    Key Points

    Question  How do clinicians conceptualize the benefits of treatments for common diseases?

    Findings  In this survey study of 542 clinicians, most respondents significantly overestimated the benefits of common therapies. Clinicians who conceptualized a greater chance of benefits of therapy were more likely to treat similar patients in their practice.

    Meaning  In this study, most clinicians were not well prepared to estimate individual patient chance of benefit, suggesting that an improved understanding of the effects of treatments could lead to more precise use of therapies and better patient outcomes.

    Abstract

    Importance  Knowing the expected effect of treatment on an individual patient is essential for patient care.

    Objective  To explore clinicians’ conceptualizations of the chance that treatments will decrease the risk of disease outcomes.

    Design, Setting, and Participants  This survey study of attending and resident physicians, nurse practitioners, and physician assistants was conducted in outpatient clinical settings in 8 US states from June 2018 to November 2019. The survey was an in-person, paper, 26-item survey in which clinicians were asked to estimate the probability of adverse disease outcomes and expected effects of therapies for diseases common in primary care.

    Main Outcomes and Measures  Estimated chance that treatments would benefit an individual patient.

    Results  Of 723 clinicians, 585 (81%) responded, and 542 completed all the questions necessary for analysis, with a median (interquartile range [IQR]) age of 32 (29-44) years, 287 (53%) women, and 294 (54%) White participants. Clinicians consistently overestimated the chance that treatments would benefit an individual patient. The median (IQR) estimated chance that warfarin would prevent a stroke in the next year was 50% (5%-80%) compared with scientific evidence, which indicates an absolute risk reduction (ARR) of 0.2% to 1.0% based on a relative risk reduction (RRR) of 39% to 50%. The median (IQR) estimated chance that antihypertensive therapy would prevent a cardiovascular event within 5 years was 30% (10%-70%) vs evidence of an ARR of 0% to 3% based on an RRR of 0% to 28%. The median (IQR) estimated chance that bisphosphonate therapy would prevent a hip fracture in the next 5 years was 40% (10%-60%) vs evidence of ARR of 0.1% to 0.4% based on an RRR of 20% to 40%. The median (IQR) estimated chance that moderate-intensity statin therapy would prevent a cardiovascular event in the next 5 years was 20% (IQR 5%-50%) vs evidence of an ARR of 0.3% to 2% based on an RRR of 19% to 33%. Estimates of the chance that a treatment would prevent an adverse outcome exceeded estimates of the absolute chance of that outcome for 60% to 70% of clinicians. Clinicians whose overestimations were greater were more likely to report using that treatment for patients in their practice (eg, use of warfarin: correlation coefficient, 0.46; 95% CI, 0.40-0.53; P < .001).

    Conclusions and Relevance  In this survey study, clinicians significantly overestimated the benefits of treatment to individual patients. Clinicians with greater overestimates were more likely to report using treatments in actual patients.

    Introduction

    Developing appropriate care plans requires that clinicians and patients have an understanding of the safety and effectiveness of medical treatments.1 Scientific evidence of the effects of treatment are generally determined by randomized clinical trials, which compare the outcomes of a disease in a group given therapy vs a control group. These therapeutic effects can be expressed in terms of the absolute or relative reduction in risk of an undesirable outcome.

    Clinicians considering prescribing or patients considering taking a therapy need to know how likely a therapy is to benefit patients individually. This can be determined by estimating the risk of an adverse outcome of disease for an individual patient and applying the expected relative risk reduction (RRR) from the therapy to identify the absolute decrement in risk for that patient, known as absolute risk reduction (ARR). While RRR is a property of a therapy across risk groups, ARR best reflects the potential benefit to an individual patient.2-8 Absolute numbers are the recommended metric for physician and patient quantification of benefits of therapies.4,9 The chance of benefit from a therapy, along with chance of harms, can then be used by clinicians and patients when considering treatment decisions.10,11 Recent public misstatements regarding the potential impact of plasma therapy for COVID-19 illustrates how conflation of RRR and ARR can be misleading. Interpreting an RRR of 35%, the Food and Drug Administration (FDA) commissioner stated that “35 out of 100 patients ‘would have been saved because of the administration of plasma.’”12 However, the ARR was actually 3%.12 Describing treatment benefits in terms of RRR provides a much larger number than ARR and is associated with patients choosing therapy.5

    Past studies have found that clinicians overestimated the benefits of treatments. However, these studies were limited by response bias, small sample size, or a focus on statistical language or technical calculations rather than realistic clinical cases.13,14 We performed a multicenter survey of clinicians in primary care practice to explore clinician estimations and communication of the expected effects of standard therapy on risk of adverse outcomes for common clinical scenarios.

    Methods
    Survey

    We developed a survey to assess clinician estimation of the expected effects of common treatments on disease outcomes and correlate estimates with reported practice. We chose scenarios in which guidelines do not give definitive recommendations and shared decision-making with patients is often recommended. The survey also included other aspects of risk perceptions, which have been presented elsewhere.15

    A draft survey was developed by primary investigators based in part on previous surveys of risk understanding.16-18 This survey was reviewed by an expert panel of clinicians with different areas of expertise, practicing in community and academic settings. The survey was further revised by the expert panel during an in-person meeting and 2 conference calls. The survey was piloted with 10 clinicians for comprehension and interpretation of questions, and minor language adjustments were made.

    Institutional review board approval was obtained from coordinating sites in Baltimore, Maryland; San Antonio, Texas; and Portland, Oregon. The institutional review boards approved this study with a waiver of informed consent because data were deidentified. The survey was performed in accordance with the American Association for Public Opinion Research (AAPOR) Standard Definitions report.19

    Clinician Risk Understanding

    The survey assessed risk estimation for common medical therapies used by primary care clinicians in routine care in situations in which treatment is informed by understanding of the probability of benefit. This was similar to previous small surveys.16-18 Individual testing questions related to treatment of atrial fibrillation, hypertension, osteoporosis, and hypercholesterolemia (eAppendix 1 in the Supplement).

    Clinicians were presented with a clinical scenario and asked to estimate the probability of adverse outcome of disease and the effect of treatment. Each scenario was created to represent a general situation but included essential details needed to estimate risk for patients (eg, age and risk factors). Responses to estimates of treatment effect on a disease outcome were compared with scientific estimates using both ARR and RRR. Additional questions were designed to assess whether errors in understanding treatment effects were associated with inaccurate estimates of the risk of adverse outcomes of the disease or a poor understanding of the reduction from treatment.

    To assess the accuracy of participant responses, we identified evidence-based estimates of the risk of disease outcome and the effect of medical therapy using a hierarchical method. We first reviewed high-quality recent systematic reviews and meta-analyses and widely accepted risk calculators. If only older systematic reviews and meta-analyses were available, with newer high-impact studies after publication, we considered data from both (attempting to understand the most accurate numbers for current technology and practice). If no systematic reviews and meta-analyses were available, we used data from studies commonly cited in recent guidelines, creating weighted averages by consensus. The expert panel of physicians overseeing the study was presented with best evidence identified, had a comment and question period, and determined consensus evidence-based answers that were used in the analysis (eAppendix 2 in the Supplement).

    Enrollment Procedure

    Each coordinating site had a primary investigator (PI) and coordinator. After obtaining institutional review board approval, we contacted the leadership of group practices or residency programs with information related to the study. Coordinators and PIs then approached practices locally and in geographically adjacent states. Investigators sought permission to give a short presentation or email introduction describing the study during a group practice meeting. Individual clinicians were then approached by a coordinator and/or PI to request participation. Lists of clinicians practicing in that clinic were reviewed. Clinicians were eligible for enrollment if they were a physician (MD or equivalent), physician assistant (PA), or nurse practitioner (NP) and cared for patients in a participating outpatient clinical area during the study dates. Clinicians were excluded if they had not seen patients in the past month.

    The survey was administered in paper format. The coordinator generally remained at the clinic, office, or meeting location until the clinician had completed the survey. If clinicians requested to complete the survey later, they were provided with an addressed, stamped envelope and could return the survey by mail, email, or by leaving it in the clinic for pick-up. Respondents were provided with a US $50 gift card for completion, if not limited by terms of employment.

    Clinicians who initially agreed to participate but did not return the survey within 2 weeks were contacted by study staff via email and/or in person as many as 5 times during a period of up to 3 months. Clinicians who did not complete the survey after these subsequent contacts were considered nonparticipants. Clinicians who declined to participate either at initial enrollment or after reminders were asked to provide a reason for not participating from a standardized list to assess for selection bias.

    Statistical Analysis

    Survey responses were entered into a REDCap database with double data entry. Data were analyzed with R version 4.0.0 (R Project for Statistical Computing) for the creation of density plots. SAS version 9.4 (SAS Institute) was used for descriptive statistics and all other statistical analyses. Comparison of those who completed all key survey questions and those who did not was done with a χ2 test. To test the correlation between estimates of benefits and clinical decisions, we used t test based on a transformation of the correlation coefficient, as implemented in SAS. A sample size of 500 was planned based on desire for generalizable results across enrollment sites. Statistical significance was set at P < .05, and all tests were 2-tailed.

    Results
    Participant Demographic Characteristics

    The survey was offered to 723 primary care physicians, NPs, and PAs practicing in Delaware, Maryland, Oregon, Pennsylvania, Texas, Virginia, Washington, and the District of Columbia (Table 1). The overall response rate was 81% (585 of 723). Of the 585 clinicians who returned the survey, we excluded 43 who did not complete all questions necessary for analysis, leaving a final sample of 542 (Table 1).

    Overall, 480 respondents (89%) were physicians with MD or DO degrees; 282 (52%) were in residency; 194 (52%) identified as White, 138 (25%) as Asian, 26 (6.6%) as Black, and 44 (8.1%) as Hispanic. The median (interquartile range) age was 32 (29-44) years, and 290 respondents (54%) were women (Table 2). The survey required a median (IQR) of 20 (15-25) minutes to complete.

    We were unable to collect any information to characterize nonresponders. We compared the 43 respondents who did not complete all necessary questions with the final cohort of 542 clinicians with complete responses. We found those not completing the survey, compared with those who did complete the survey, were more likely to be women (31 [72%] vs 290 [54%]; P = .02), to be residents (13 [30%] vs 282 [52%]; P = .006), and to be NPs or PAs (12 [29%] vs 62 [11%]; P = .001).

    Estimates of Adverse Outcomes of Disease and Effects of Treatment

    Clinician estimates of the probability of benefit from medical therapy were consistently higher than scientific evidence for ARR or RRR (Figure 1). Estimates did not significantly differ between trainees and clinicians in practice (eTable in the Supplement).

    In the atrial fibrillation clinical scenario, the median (IQR) estimated probability of stroke with atrial fibrillation without treatment was 5% (2%-15%), while evidence supported a rate of 0.4% to 2%. The median (IQR) estimated chance that warfarin would prevent a stroke in that patient in the next year was 50% (5%-80%) compared with evidence revealing an ARR of 0.2% to 1% from an RRR of 39% to 50% for stroke prevention. Clinician estimates of benefits from therapy exceeded the total estimated chance of an adverse disease outcome in 379 responses (70%). The estimated benefits of therapy exceeded the upper limit of RRR for 235 clinicians (43%).

    In the mild hypertension scenario, the median (IQR) estimated probability of a cardiovascular event without treatment was 10% (6%-20%), while evidence supported a rate of 3% to 12%. The median (IQR) estimated chance that antihypertensive therapy would prevent a cardiovascular event in that patient within 5 years was 30% (10%-70%), while evidence supported an ARR of 0% to 3.3% from an RRR of 0% to 28%. Clinician estimates of benefits from therapy exceeded the total estimated chance of an adverse disease outcome for 340 responses (63%). The estimated benefits of therapy exceeded the upper limit of RRR for 294 clinicians (54%).

    In the osteoporosis scenario, the median (IQR) estimated probability of hip fracture without treatment was 10% (5%-25%), while evidence supported a rate of 0.3% to 1%. The median (IQR) estimated chance that bisphosphonate therapy would prevent a hip fracture in that patient in the next 5 years was 40% (10%-60%) compared with evidence expressed most accurately as an ARR of 0.1% to 0.4% from an RRR of 20% to 40%. Clinician estimates of benefits from therapy exceeded the total estimated chance of an adverse disease outcome for 343 responses (63%). The estimated benefits of therapy exceeded the upper limit of the RRR for 248 clinicians (46%).

    In the mild hyperlipidemia scenario, the median (IQR) estimated probability of cardiovascular event without treatment was 10% (5%-15%), while evidence supported a rate of 2% to 5%. The median (IQR) estimated chance that moderate-intensity statin therapy would prevent a cardiovascular event in that patient in the next 5 years was 20% (5%-50%) vs evidence of an ARR of 0.3% to 2% from an RRR of 19% to 33%. Clinician estimates of benefits from therapy exceeded the total estimated chance of an adverse disease outcome for 323 responses (60%). The estimated benefits of therapy exceeded the upper limit of the RRR for 204 clinicians (38%).

    A hypothetical scenario addressing test interpretation revealed better understanding than with clinically grounded questions. For the question, “Consider a condition in which 3% of patients will develop a bad outcome within 5 years. Treatment has a relative risk reduction of 33%. If 100 patients with this condition are treated for 5 years, how many patients will have a bad outcome prevented by treatment?” the median (IQR) response was 2 (1-20) of 100 patients. (The correct answer was 1 of 100 patients.)

    Association Between Estimates of Treatment Effects and Recommending Treatment to Patients

    To evaluate whether estimates of benefits of medical therapy correlated with clinical practice decisions, we compared 2 answers from each respondent: (1) “What will you tell [a specific patient] is the chance that [a treatment] will prevent them from having [a specific disease outcome]?” and (2) “In your practice, in what proportion of patients with similar risk do you recommend treatment?” We found a moderate positive correlation for each treatment scenario between estimates of the degree to which treatment prevented a disease outcome and how often they recommended such treatment in their real-life patients (eg, use of warfarin: correlation coefficient, 0.46; 95% CI, 0.40-0.53; P < .001) (Figure 2). For the scenario involving atrial fibrillation, clinicians answering near the ARR range of 1% chance of preventing a stroke in this patient recommended treatment in approximately 35% of similar patients in their practice, while the median clinician estimated a 50% chance of benefit and reported treating approximately 60% of similar patients in their practice. Similar effects were seen with other scenarios.

    Discussion

    In scenarios routinely encountered in practice, clinicians significantly overestimated the benefits of medical therapy. When asked, “What will you tell [a specific patient] is the chance that [a treatment] will prevent them from having [a specific disease outcome]?” clinicians rarely responded with estimates in the range of the ARR, as recommended for clinical decision-making. Most clinicians estimated individual benefits in the range of RRR, and one-third to one-half estimated benefits beyond the RRR. Clinicians who overestimated benefits of treatment were more likely to report recommending that treatment to similar patients in their own practice.

    Few clinicians in our study appeared to use ARR to anchor estimates of individual patient benefit. ARR is the best understood and most helpful metric of impact of therapy for an individual patient,20,21 but it is often small compared with RRR. Not surprisingly, RRR is often preferentially reported in the literature for research with small effect sizes or by the pharmaceutical industry, practices that have been criticized.22-24 Clinicians do not receive extensive training or a consistent method to consider potential benefits and harms of therapy for individual patients, and many clinicians learn treatment approaches from yes/no–type guideline recommendations or consider benefits based on clinical experience, physiological thinking, or other theoretical models.25-27

    We are uncertain whether clinicians conceptualize patient benefit using evidence-based medicine metrics. It is clear clinicians rarely conceptualize benefit in terms of ARR. Their estimates suggest conceptualization of individual benefit to a patient in general terms closer to RRR and not ARR. Most respondents estimated the chance that the patient would benefit from a treatment to be a greater than the total chance that the same patient would have a negative outcome from disease. This response is only logically consistent if outcomes are conceptualized in absolute chance and benefits in relative terms. For example, in the atrial fibrillation scenario, the average clinician estimated that for a specific patient, Mr Miller, the likelihood of stroke was 5% and the chance that warfarin therapy would prevent Mr Miller from having a stroke was 50%. This is despite the survey question posing, “What would you tell Mr. Miller is the chance that warfarin will prevent him from having a stroke in the next year?” For the questions in our survey, the 20% to 50% estimated chance of benefit from common treatments to a patient would dramatically inflate expected benefit compared with the recommended ARR, which ranged across examples from 1% to 3%.

    Notably, in our study, overestimates of treatment benefit often went beyond RRR. For each question, one-third to one-half of estimates of benefit were beyond the upper end of the RRR. This would imply that many clinicians either grossly overestimated the RRR for common treatments or that they did not conceptualize patient benefit from therapy in the evidence-based medicine approach of using language of chance or probability.

    In contrast to the clinical scenarios presented, in 1 hypothetical question, clinicians correctly calculated ARR from RRR. The ability to technically calculate ARR while reverting to the much higher estimates of RRR to understand commonly prescribed treatments implies evidence-based principles of treatment may be understood theoretically but are not understood or applied to clinical care.28 The observed overestimates of benefit are also consistent with cognitive biases that have been described on other contexts, including base rate neglect, anchoring bias, and confirmation bias,29-31 as well as the observation that humans often exaggerate small risks.32

    Our survey presented clinical scenarios that are common in primary care. For the patients described, guidelines would generally not recommend treatment for atrial fibrillation33 or osteoporosis34; treatment of hypertension and hypercholesterolemia could possibly be recommended if unreported risk factors were present.35,36 We found that in these patient scenarios, physicians who overestimated treatment benefit were more likely to report treating similar patients in their own practice. This suggests that overestimates of treatment benefit may directly affect clinical decisions, potentially exposing patients to therapies for which potential harms outweigh benefits. This matches studies finding frequent overuse of treatment for low-risk patients, such as in atrial fibrillation.37

    Clinician overestimates of treatment benefits would severely limit shared decision-making. Shared decision-making is broadly recommended38,39 and is often invoked as a key component for improving care.40-42 When clinicians engage in shared decision-making, there is a general consensus that patients should be presented with harms and benefits, using absolute and not relative numbers.1,2,11,22,43 Training in shared decision-making has focused on communication skills but may also require training on probability of benefit from treatment.44 While the optimal approach to improve these skills has not been described, educational approaches, risk calculators, decision aids, and behavioral nudges may be helpful.

    Limitations

    This study has limitations, including that the small fraction of respondents who did not complete the survey were more likely to be women, NPs, or PAs. However, the overall response rate was high. While validity was extensively assessed via a multidisciplinary expert panel, reliability of our novel survey was not assessed. We requested estimates of benefits and interpreted conceptualization from answers provided. This gave more concrete answers but did not fully assess how treatment benefit was conceptualized by individual clinicians and may not capture the totality of communication in a clinical encounter.

    Conclusions

    In this study, clinicians performing primary care significantly overestimated the chance that common therapies would benefit individual patients. Clinicians with the largest overestimates were also likely to prescribe these treatments in patients in their practice. Widespread overestimates of the benefits of therapy likely contribute to overtreatment in actual patients. Improved understanding of the effects of treatments is essential for precise use of therapies and better patient outcomes.

    Back to top
    Article Information

    Accepted for Publication: April 29, 2021.

    Published: July 21, 2021. doi:10.1001/jamanetworkopen.2021.19747

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Morgan DJ et al. JAMA Network Open.

    Corresponding Author: Daniel J. Morgan, MD, MS, Department of Epidemiology and Public Health, University of Maryland School of Medicine, 10 S Pine St, MSTF 334, Baltimore, MD 21201 (dmorgan@som.umaryland.edu).

    Author Contributions: Dr Morgan had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Morgan, Owczarzak, Magder, Scherer, Brown, Pfeiffer, Terndrup, Feldstein, Foy, Masnick, Weisenberg, Korenstein.

    Acquisition, analysis, or interpretation of data: Morgan, Pineles, Owczarzak, Magder, Scherer, Pfeiffer, Terndrup, Leykum, Feldstein, Stevens, Koch, Korenstein.

    Drafting of the manuscript: Pineles, Magder, Terndrup, Stevens, Weisenberg.

    Critical revision of the manuscript for important intellectual content: Morgan, Owczarzak, Magder, Scherer, Brown, Pfeiffer, Terndrup, Leykum, Feldstein, Foy, Koch, Masnick, Weisenberg, Korenstein.

    Statistical analysis: Morgan, Magder.

    Obtained funding: Morgan.

    Administrative, technical, or material support: Morgan, Pineles, Owczarzak, Scherer, Terndrup, Stevens, Koch.

    Supervision: Morgan.

    Conflict of Interest Disclosures: Dr Morgan reported receiving grants from Veterans Affairs, the Agency for Healthcare Research and Quality, and the US Centers for Disease Control and Prevention and receiving travel reimbursement for conference speaking or coordination from the Infectious Diseases Society of America, Society for Healthcare Epidemiology of America, and the Lown Institute outside the submitted work. Dr Scherer reported receiving grants from the Agency for Healthcare Research and Quality during the conduct of the study. Dr Brown reported receiving grants from the National Institutes of Health outside the submitted work. Dr Pfeiffer reported receiving grants from Pfizer outside the submitted work. Dr Feldstein reported receiving an honorarium from the University of Maryland–Baltimore during the conduct of the study. Dr Korenstein reported that their spouse serves on the scientific advisory board of and has equity interest in Vedanta Biosciences, serves on the scientific advisory board of the Pandemic Response Lab–NYC, and provides consulting for Fibrion. No other disclosures were reported.

    Funding/Support: This project was funded by New Innovator Award DP2LM012890 from the National Institutes of Health National Library of Medicine to Dr Morgan.

    Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Additional Contributions: We would like to acknowledge all the primary care practitioners who generously volunteered their time to take part in this study.

    References
    1.
    US Food and Drug Administration. The FDA’s drug review process: ensuring drugs are safe and effective. November 24, 2017. Accessed September 21, 2020. https://www.fda.gov/drugs/drug-information-consumers/fdas-drug-review-process-ensuring-drugs-are-safe-and-effective
    2.
    Armstrong  KA, Metlay  JP.  Annals clinical decision making: translating population evidence to individual patients.   Ann Intern Med. 2020;172(9):610-616. doi:10.7326/M19-3496PubMedGoogle ScholarCrossref
    3.
    Elliott  MH, Skydel  JJ, Dhruva  SS, Ross  JS, Wallach  JD.  Characteristics and reporting of number needed to treat, number needed to harm, and absolute risk reduction in controlled clinical trials, 2001-2019.   JAMA Intern Med. 2021;181(2):282-284. doi:10.1001/jamainternmed.2020.4799PubMedGoogle ScholarCrossref
    4.
    Zipkin  DA, Umscheid  CA, Keating  NL,  et al.  Evidence-based risk communication: a systematic review.   Ann Intern Med. 2014;161(4):270-280. doi:10.7326/M14-0295PubMedGoogle ScholarCrossref
    5.
    Gigerenzer  G.  Why does framing influence judgment?   J Gen Intern Med. 2003;18(11):960-961. doi:10.1046/j.1525-1497.2003.30901.xPubMedGoogle ScholarCrossref
    6.
    Paling  J.  Strategies to help patients understand risks.   BMJ. 2003;327(7417):745-748. doi:10.1136/bmj.327.7417.745PubMedGoogle ScholarCrossref
    7.
    Barratt  A, Trevena  L, Davey  HM, McCaffery  K.  Use of decision aids to support informed choices about screening.   BMJ. 2004;329(7464):507-510. doi:10.1136/bmj.329.7464.507PubMedGoogle ScholarCrossref
    8.
    Nuovo  J, Melnikow  J, Chang  D.  Reporting number needed to treat and absolute risk reduction in randomized controlled trials.   JAMA. 2002;287(21):2813-2814. doi:10.1001/jama.287.21.2813PubMedGoogle ScholarCrossref
    9.
    Epstein  D; ProPublica. When evidence says no, but doctors say yes. The Atlantic. February 22, 2017. Accessed August 18, 2020. https://www.theatlantic.com/health/archive/2017/02/when-evidence-says-no-but-doctors-say-yes/517368/
    10.
    Armstrong  KA, Metlay  JP.  Annals clinical decision making: communicating risk and engaging patients in shared decision making.   Ann Intern Med. 2020;172(10):688-692. doi:10.7326/M19-3495PubMedGoogle ScholarCrossref
    11.
    Morgan  DJ, Scherer  LD, Korenstein  D.  Improving physician communication about treatment decisions: reconsideration of “risks vs benefits.”   JAMA. 2020;324(10):937-938. doi:10.1001/jama.2020.0354PubMedGoogle ScholarCrossref
    12.
    Thomas  K, Fink  S. FDA “grossly misrepresented” blood plasma data, scientists say. The New York Times. August 24, 2020. Accessed September 21, 2020. https://www.nytimes.com/2020/08/24/health/fda-blood-plasma.html
    13.
    Hoffmann  TC, Del Mar  C.  Clinicians’ expectations of the benefits and harms of treatments, screening, and tests: a systematic review.   JAMA Intern Med. 2017;177(3):407-419. doi:10.1001/jamainternmed.2016.8254PubMedGoogle ScholarCrossref
    14.
    Treadwell  JS, Wong  G, Milburn-Curtis  C, Feakins  B, Greenhalgh  T.  GPs’ understanding of the benefits and harms of treatments for long-term conditions: an online survey.   BJGP Open. 2020;4(1):bjgpopen20X101016. doi:10.3399/bjgpopen20X101016PubMedGoogle Scholar
    15.
    Morgan  DJ, Pineles  L, Owczarzak  J,  et al.  Accuracy of practitioner estimates of probability of diagnosis before and after testing.   JAMA Intern Med. 2021;181(6):747-755. doi:10.1001/jamainternmed.2021.0269PubMedGoogle ScholarCrossref
    16.
    Gigerenzer  G.  Reckoning With Risk: Learning to Live with Uncertainty. Penguin Books; 2003.
    17.
    Krouss  M, Croft  L, Morgan  DJ.  Physician understanding and ability to communicate harms and benefits of common medical treatments.   JAMA Intern Med. 2016;176(10):1565-1567. doi:10.1001/jamainternmed.2016.5027PubMedGoogle ScholarCrossref
    18.
    Fagerlin  A, Sepucha  KR, Couper  MP, Levin  CA, Singer  E, Zikmund-Fisher  BJ.  Patients’ knowledge about 9 common health conditions: the DECISIONS survey.   Med Decis Making. 2010;30(5)(suppl):35S-52S. doi:10.1177/0272989X10378700PubMedGoogle Scholar
    19.
    The American Association for Public Opinion Research.  Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 9th edition. AAPOR; 2016.
    20.
    Woloshin  S, Schwartz  LM.  Communicating data about the benefits and harms of treatment: a randomized trial.   Ann Intern Med. 2011;155(2):87-96. doi:10.7326/0003-4819-155-2-201107190-00004PubMedGoogle ScholarCrossref
    21.
    Barratt  A, Wyer  PC, Hatala  R,  et al; Evidence-Based Medicine Teaching Tips Working Group.  Tips for learners of evidence-based medicine: 1. relative risk reduction, absolute risk reduction and number needed to treat.   CMAJ. 2004;171(4):353-358. doi:10.1503/cmaj.1021197PubMedGoogle ScholarCrossref
    22.
    Guyatt  GH, Oxman  AD, Vist  GE,  et al; GRADE Working Group.  GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.   BMJ. 2008;336(7650):924-926. doi:10.1136/bmj.39489.470347.ADPubMedGoogle ScholarCrossref
    23.
    World Health Organization/Health Action International. Understanding and responding to pharmaceutical promotion–a practical guide. Accessed December 16, 2020. https://www.paho.org/hq/dmdocuments/2011/drug-promotion-manual-CAP-3-090610.pdf
    24.
    Lexchin  J.  How patient outcomes are reported in drug advertisements.   Can Fam Physician. 1999;45:1213-1216.PubMedGoogle Scholar
    25.
    Rodman  A,. Episode 50: I know nothing. Bedside Rounds. October 27, 2019. Accessed November 24, 2020. http://bedside-rounds.org/episode-50-i-know-nothing/
    26.
    Prasad  V, Ho  N.  Why do we continue to adopt medical practices based on pathophysiology alone when we should be insisting on clinical trials?   J Clin Epidemiol. 2014;67(4):361-363. doi:10.1016/j.jclinepi.2013.11.009PubMedGoogle ScholarCrossref
    27.
    Montgomery  K.  How Doctors Think: Clinical Judgment and the Practice of Medicine. Oxford University Press; 2005.
    28.
    Khan  KS, Coomarasamy  A.  A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine.   BMC Med Educ. 2006;6:59. doi:10.1186/1472-6920-6-59PubMedGoogle ScholarCrossref
    29.
    Tversky  A, Kahneman  D.  Judgment under uncertainty: heuristics and biases.   Science. 1974;185(4157):1124-1131. doi:10.1126/science.185.4157.1124PubMedGoogle ScholarCrossref
    30.
    Saposnik  G, Redelmeier  D, Ruff  CC, Tobler  PN.  Cognitive biases associated with medical decisions: a systematic review.   BMC Med Inform Decis Mak. 2016;16(1):138. doi:10.1186/s12911-016-0377-1PubMedGoogle ScholarCrossref
    31.
    Crowley  RS, Legowski  E, Medvedeva  O,  et al.  Automated detection of heuristics and biases among pathologists in a computer-based system.   Adv Health Sci Educ Theory Pract. 2013;18(3):343-363. doi:10.1007/s10459-012-9374-zPubMedGoogle ScholarCrossref
    32.
    Kusev  P, van Schaik  P, Ayton  P, Dent  J, Chater  N.  Exaggerated risk: prospect theory and probability weighting in risky choice.   J Exp Psychol Learn Mem Cogn. 2009;35(6):1487-1505. doi:10.1037/a0017039PubMedGoogle ScholarCrossref
    33.
    DeJong  J. CHA2DS2-VASc/HAS-BLED/EHRA atrial fibrillation risk score calculator. Accessed June 16, 2021. https://www.chadsvasc.org/
    34.
    Viswanathan  M, Reddy  S, Berkman  N,  et al.  Screening to prevent osteoporotic fractures: updated evidence report and systematic review for the US Preventive Services Task Force.   JAMA. 2018;319(24):2532-2551. doi:10.1001/jama.2018.6537PubMedGoogle ScholarCrossref
    35.
    Whelton  PK, Carey  RM, Aronow  WS,  et al.  2017 ACC/AHA/AAPA/ABC/ACPM/AGS/APhA/ASH/ASPC/NMA/PCNA guideline for the prevention, detection, evaluation, and management of high blood pressure in adults: a report of the American College of Cardiology/American Heart Association task force on clinical practice guidelines.   Hypertension. 2018;71(6):e13-e115. doi:10.1161/HYP.0000000000000065PubMedGoogle Scholar
    36.
    Whelton  PK, Williams  B.  The 2018 European Society of Cardiology/European Society of Hypertension and 2017 American College of Cardiology/American Heart Association blood pressure guidelines: more similar than different.   JAMA. 2018;320(17):1749-1750. doi:10.1001/jama.2018.16755PubMedGoogle ScholarCrossref
    37.
    Morgan  DJ, Dhruva  SS, Wright  SM, Korenstein  D.  2016 Update on medical overuse: a systematic review.   JAMA Intern Med. 2016;176(11):1687-1692. doi:10.1001/jamainternmed.2016.5381PubMedGoogle ScholarCrossref
    38.
    Wells’ Criteria for Pulmonary Embolism. MDCalc. Accessed April 27, 2020. https://www.mdcalc.com/wells-criteria-pulmonary-embolism
    39.
    January  CT, Wann  LS, Calkins  H,  et al.  2019 AHA/ACC/HRS focused update of the 2014 AHA/ACC/HRS guideline for the management of patients with atrial fibrillation: a report of the American College of Cardiology/American Heart Association task force on clinical practice guidelines and the Heart Rhythm Society in collaboration with the Society of Thoracic Surgeons.   Circulation. 2019;140(2):e125-e151. doi:10.1161/CIR.0000000000000665PubMedGoogle ScholarCrossref
    40.
    Sheridan  SL, Donahue  KE, Brenner  AT.  Beginning with high value care in mind: a scoping review and toolkit to support the content, delivery, measurement, and sustainment of high value care.   Patient Educ Couns. 2019;102(2):238-252. doi:10.1016/j.pec.2018.05.014PubMedGoogle ScholarCrossref
    41.
    Morgan  DJ, Leppin  AL, Smith  CD, Korenstein  D.  A practical framework for understanding and reducing medical overuse: conceptualizing overuse through the patient-clinician interaction.   J Hosp Med. 2017;12(5):346-351. doi:10.12788/jhm.2738PubMedGoogle ScholarCrossref
    42.
    Grundy  SM, Stone  NJ, Bailey  AL,  et al.  2018 AHA/ACC/AACVPR/AAPA/ABC/ACPM/ADA/AGS/APhA/ASPC/NLA/PCNA guideline on the management of blood cholesterol: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines.   J Am Coll Cardiol. 2019;73(24):e285-e350. doi:10.1016/j.jacc.2018.11.003PubMedGoogle ScholarCrossref
    43.
    Sheridan  SL, Pignone  MP, Lewis  CL.  A randomized comparison of patients’ understanding of number needed to treat and other common risk reduction formats.   J Gen Intern Med. 2003;18(11):884-892. doi:10.1046/j.1525-1497.2003.21102.xPubMedGoogle ScholarCrossref
    44.
    Back  AL, Fromme  EK, Meier  DE.  Training clinicians with communication skills needed to match medical treatments to patient values.   J Am Geriatr Soc. 2019;67(S2):S435-S441. doi:10.1111/jgs.15709PubMedGoogle ScholarCrossref
    ×