Effect of Integrating Machine Learning Mortality Estimates With Behavioral Nudges to Clinicians on Serious Illness Conversations Among Patients With Cancer: A Stepped-Wedge Cluster Randomized Clinical Trial | Oncology | JAMA Oncology | JAMA Network
[Skip to Content]
[Skip to Content Landing]
Figure.  CONSORT Diagram
CONSORT Diagram

SIC indicates serious illness conversation.

Table 1.  Patient-Encounter Characteristics
Patient-Encounter Characteristics
Table 2.  Adjusted Changes in Serious Illness Conversations and in Advanced Care Planning
Adjusted Changes in Serious Illness Conversations and in Advanced Care Planning
1.
Emanuel  EJ, Young-Xu  Y, Levinsky  NG, Gazelle  G, Saynina  O, Ash  AS.  Chemotherapy use among Medicare beneficiaries at the end of life.   Ann Intern Med. 2003;138(8):639-643. doi:10.7326/0003-4819-138-8-200304150-00011 PubMedGoogle ScholarCrossref
2.
Earle  CC, Neville  BA, Landrum  MB, Ayanian  JZ, Block  SD, Weeks  JC.  Trends in the aggressiveness of cancer care near the end of life.   J Clin Oncol. 2004;22(2):315-321. doi:10.1200/JCO.2004.08.136 PubMedGoogle ScholarCrossref
3.
Earle  CC, Landrum  MB, Souza  JM, Neville  BA, Weeks  JC, Ayanian  JZ.  Aggressiveness of cancer care near the end of life: is it a quality-of-care issue?   J Clin Oncol. 2008;26(23):3860-3866. doi:10.1200/JCO.2007.15.8253 PubMedGoogle ScholarCrossref
4.
Chastek  B, Harley  C, Kallich  J, Newcomer  L, Paoli  CJ, Teitelbaum  AH.  Health care costs for patients with cancer at the end of life.   J Oncol Pract. 2012;8(6):75s-80s. doi:10.1200/JOP.2011.000469 PubMedGoogle ScholarCrossref
5.
Wen  F-H, Chen  J-S, Su  P-J,  et al.  Terminally ill cancer patients’ concordance between preferred life-sustaining treatment states in their last six months of life and received life-sustaining treatment states in their last month: an observational study.   J Pain Symptom Manage. 2018;56(4):509-518.e3. doi:10.1016/j.jpainsymman.2018.07.003 PubMedGoogle ScholarCrossref
6.
Bernacki  RE, Block  SD; American College of Physicians High Value Care Task Force.  Communication about serious illness care goals: a review and synthesis of best practices.   JAMA Intern Med. 2014;174(12):1994-2003. doi:10.1001/jamainternmed.2014.5271 PubMedGoogle ScholarCrossref
7.
Wright  AA, Zhang  B, Ray  A,  et al.  Associations between end-of-life discussions, patient mental health, medical care near death, and caregiver bereavement adjustment.   JAMA. 2008;300(14):1665-1673. doi:10.1001/jama.300.14.1665 PubMedGoogle ScholarCrossref
8.
Brinkman-Stoppelenburg  A, Rietjens  JA, van der Heide  A.  The effects of advance care planning on end-of-life care: a systematic review.   Palliat Med. 2014;28(8):1000-1025. doi:10.1177/0269216314526272 PubMedGoogle ScholarCrossref
9.
Schubart  JR, Levi  BH, Bain  MM, Farace  E, Green  MJ.  Advance care planning among patients with advanced cancer.   J Oncol Pract. 2019;15(1):e65-e73. doi:10.1200/JOP.18.00044 PubMedGoogle ScholarCrossref
10.
National Quality Forum: Palliative and end-of-life care 2015-2016. Draft report for comment June 20, 2016. Accessed June 10, 2020. http://www.qualityforum.org/Projects/n-r/Palliative_and_End-of-Life_Care_Project_2015-2016/Draft_Report_for_Comment.aspx
11.
Chow  E, Harth  T, Hruby  G, Finkelstein  J, Wu  J, Danjoux  C.  How accurate are physicians’ clinical predictions of survival and the available prognostic tools in estimating survival times in terminally ill cancer patients? a systematic review.   Clin Oncol (R Coll Radiol). 2001;13(3):209-218. doi:10.1007/s001740170078 PubMedGoogle Scholar
12.
Ermacora  P, Mazzer  M, Isola  M,  et al.  Prognostic evaluation in palliative care: final results from a prospective cohort study.   Support Care Cancer. 2019;27(6):2095-2102. doi:10.1007/s00520-018-4463-z PubMedGoogle ScholarCrossref
13.
Weeks  JC, Cook  EF, O’Day  SJ,  et al.  Relationship between cancer patients’ predictions of prognosis and their treatment preferences.   JAMA. 1998;279(21):1709-1714. doi:10.1001/jama.279.21.1709 PubMedGoogle ScholarCrossref
14.
Rose  JH, O’Toole  EE, Dawson  NV,  et al.  Perspectives, preferences, care practices, and outcomes among older and middle-aged patients with late-stage cancer.   J Clin Oncol. 2004;22(24):4907-4917. doi:10.1200/JCO.2004.06.050 PubMedGoogle ScholarCrossref
15.
White  N, Reid  F, Harris  A, Harries  P, Stone  P.  A systematic review of predictions of survival in palliative care: how accurate are clinicians and who are the experts?   PLoS One. 2016;11(8):e0161407. doi:10.1371/journal.pone.0161407 PubMedGoogle Scholar
16.
Brown  B, Gude  WT, Blakeman  T,  et al.  Clinical Performance Feedback Intervention Theory (CP-FIT): a new theory for designing, implementing, and evaluating feedback in health care based on a systematic review and meta-synthesis of qualitative research.   Implement Sci. 2019;14(1):40. doi:10.1186/s13012-019-0883-5 PubMedGoogle ScholarCrossref
17.
Navathe  AS, Emanuel  EJ.  Physician peer comparisons as a nonfinancial strategy to improve the value of care.   JAMA. 2016;316(17):1759-1760. doi:10.1001/jama.2016.13739 PubMedGoogle ScholarCrossref
18.
Patel  MS, Volpp  KG, Asch  DA.  Nudge units to improve the delivery of health care.   N Engl J Med. 2018;378(3):214-216. doi:10.1056/NEJMp1712984 PubMedGoogle ScholarCrossref
19.
Patel  MS, Volpp  KG, Small  DS,  et al.  Using active choice within the electronic health record to increase influenza vaccination rates.   J Gen Intern Med. 2017;32(7):790-795. doi:10.1007/s11606-017-4046-6 PubMedGoogle ScholarCrossref
20.
Bernacki  R, Hutchings  M, Vick  J,  et al.  Development of the Serious Illness Care Program: a randomised controlled trial of a palliative care communication intervention.   BMJ Open. 2015;5(10):e009032. doi:10.1136/bmjopen-2015-009032 PubMedGoogle Scholar
21.
Paladino  J, Bernacki  R, Neville  BA,  et al.  Evaluating an intervention to improve communication between oncology clinicians and patients with life-limiting cancer: a cluster randomized clinical trial of the Serious Illness Care Program.   JAMA Oncol. 2019;5(6):801-809. doi:10.1001/jamaoncol.2019.0292 PubMedGoogle ScholarCrossref
22.
Bernacki  R, Paladino  J, Neville  BA,  et al.  Effect of the Serious Illness Care Program in outpatient oncology: a cluster randomized clinical trial.   JAMA Intern Med. 2019;179(6):751-759. doi:10.1001/jamainternmed.2019.0077 PubMedGoogle ScholarCrossref
23.
Parikh  RB, Manz  C, Chivers  C,  et al.  Machine learning approaches to predict 6-month mortality among patients with cancer.   JAMA Netw Open. 2019;2(10):e1915997. doi:10.1001/jamanetworkopen.2019.15997 PubMedGoogle Scholar
24.
Machine-generated mortality estimates and nudges to promote advance care planning discussion among cancer patients. ClinicalTrials.gov identifier: NCT03984773. Accessed September 2, 2020. https://clinicaltrials.gov/ct2/show/NCT03984773
25.
Manz  CR, Parikh  RB, Evans  CN,  et al.  Integrating machine-generated mortality estimates and behavioral nudges to promote serious illness conversations for cancer patients: design and methods for a stepped-wedge cluster randomized controlled trial.   Contemp Clin Trials. 2020;90:105951. doi:10.1016/j.cct.2020.105951 PubMedGoogle Scholar
26.
Patel  MS, Kurtzman  GW, Kannan  S,  et al.  Effect of an automated patient dashboard using active choice and peer comparison performance feedback to physicians on statin prescribing: the prescribe cluster randomized clinical trial.   JAMA Netw Open. 2018;1(3):e180818. doi:10.1001/jamanetworkopen.2018.0818 PubMedGoogle Scholar
27.
Meeker  D, Linder  JA, Fox  CR,  et al.  Effect of behavioral interventions on inappropriate antibiotic prescribing among primary care practices: a randomized clinical trial.   JAMA. 2016;315(6):562-570. doi:10.1001/jama.2016.0275 PubMedGoogle ScholarCrossref
28.
Halpern  SD, Ubel  PA, Asch  DA.  Harnessing the power of default options to improve health care.   N Engl J Med. 2007;357(13):1340-1344. doi:10.1056/NEJMsb071595PubMedGoogle ScholarCrossref
29.
Patel  MS, Day  SC, Halpern  SD,  et al.  Generic medication prescription rates after health system–wide redesign of default options within the electronic health record.   JAMA Intern Med. 2016;176(6):847-848. doi:10.1001/jamainternmed.2016.1691 PubMedGoogle ScholarCrossref
30.
Patel  MS, Day  S, Small  DS,  et al.  Using default options within the electronic health record to increase the prescribing of generic-equivalent medications: a quasi-experimental study.   Ann Intern Med. 2014;161(10)(suppl):S44-S52. doi:10.7326/M13-3001PubMedGoogle ScholarCrossref
31.
Sharma  S, Guttmann  D, Small  DS,  et al.  Effect of introducing a default order in the electronic medical record on unnecessary daily imaging during palliative radiotherapy for adults with cancer: a stepped-wedge cluster randomized clinical trial.   JAMA Oncol. 2019;5(8):1220. doi:10.1001/jamaoncol.2019.1432 PubMedGoogle ScholarCrossref
32.
Delgado  MK, Shofer  FS, Patel  MS,  et al.  Association between electronic medical record implementation of default opioid prescription quantities and prescribing behavior in two emergency departments.   J Gen Intern Med. 2018;33(4):409-411. doi:10.1007/s11606-017-4286-5 PubMedGoogle ScholarCrossref
33.
Efron  B, Tibshirani  R.  An Introduction to the Bootstrap. New York, NY: Chapman & Hall; 1993. doi:10.1007/978-1-4899-4541-9
34.
Davison  AC, Hinkley  DV.  Bootstrap Methods and Their Application. New York, NY: Cambridge University Press; 1997. doi:10.1017/CBO9780511802843
35.
Schofield  G, Kreeger  L, Meyer  M,  et al.  Implementation of a quality improvement programme to support advance care planning in five hospitals across a health region.   BMJ Support Palliat Care. 2015;5(1):91-94. doi:10.1136/bmjspcare-2013-000574 PubMedGoogle ScholarCrossref
36.
Temel  JS, Greer  JA, Gallagher  ER,  et al.  Electronic prompt to improve outpatient code status documentation for patients with advanced lung cancer.   J Clin Oncol. 2013;31(6):710-715. doi:10.1200/JCO.2012.43.2203 PubMedGoogle ScholarCrossref
37.
Bestvina  CM, Polite  BN.  Implementation of advance care planning in oncology: a review of the literature.   J Oncol Pract. 2017;13(10):657-662. doi:10.1200/JOP.2017.021246 PubMedGoogle ScholarCrossref
38.
Titano  JJ, Badgeley  M, Schefflein  J,  et al.  Automated deep-neural-network surveillance of cranial images for acute neurologic events.   Nat Med. 2018;24(9):1337-1341. doi:10.1038/s41591-018-0147-y PubMedGoogle ScholarCrossref
39.
Lin  H, Li  R, Liu  Z,  et al.  Diagnostic efficacy and therapeutic decision-making capacity of an artificial intelligence platform for childhood cataracts in eye clinics: a multicentre randomized controlled trial.   EClinicalMedicine. 2019;9:52-59. doi:10.1016/j.eclinm.2019.03.001 PubMedGoogle ScholarCrossref
40.
Emanuel  EJ, Ubel  PA, Kessler  JB,  et al.  Using behavioral economics to design physician incentives that deliver high-value care.   Ann Intern Med. 2016;164(2):114-119. doi:10.7326/M15-1330 PubMedGoogle ScholarCrossref
41.
LeBlanc  TW, Back  AL, Danis  M, Abernethy  AP.  Electronic health records (EHRs) in the oncology clinic: how clinician interaction with EHRs can improve communication with the patient.   J Oncol Pract. 2014;10(5):317-321. doi:10.1200/JOP.2014.001385 PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Views 3,620
    Citations 0
    Original Investigation
    October 15, 2020

    Effect of Integrating Machine Learning Mortality Estimates With Behavioral Nudges to Clinicians on Serious Illness Conversations Among Patients With Cancer: A Stepped-Wedge Cluster Randomized Clinical Trial

    Author Affiliations
    • 1Department of Population Sciences, Dana-Farber Cancer Institute, Boston, Massachusetts
    • 2Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia
    • 3Penn Center for Cancer Care Innovation, Abramson Cancer Center, University of Pennsylvania, Philadelphia
    • 4Corporal Michael J. Crescenz Veterans Affairs Medical Center, Philadelphia, Pennsylvania
    • 5Wharton School of the University of Pennsylvania, Philadelphia
    • 6Penn Medicine Nudge Unit, Philadelphia, Pennsylvania
    • 7University of Pennsylvania Health System, Philadelphia
    JAMA Oncol. Published online October 15, 2020. doi:10.1001/jamaoncol.2020.4759
    Key Points

    Question  What is the effect of delivering machine learning mortality predictions with behavioral nudges to oncology clinicians on the rate of serious illness conversations with patients with cancer?

    Findings  In this stepped-wedge cluster randomized clinical trial that included 14 607 patients with cancer, the intervention led to a significant increase in serious illness conversations from approximately 1% to 5% of all patient encounters and from approximately 4% to 15% of encounters with patients having high predicted mortality risk.

    Meaning  Machine learning mortality predictions combined with behavioral nudges to clinicians led to an increased rate of serious illness conversations for patients with cancer.

    Abstract

    Importance  Serious illness conversations (SICs) are structured conversations between clinicians and patients about prognosis, treatment goals, and end-of-life preferences. Interventions that increase the rate of SICs between oncology clinicians and patients may improve goal-concordant care and patient outcomes.

    Objective  To determine the effect of a clinician-directed intervention integrating machine learning mortality predictions with behavioral nudges on motivating clinician-patient SICs.

    Design, Setting, and Participants  This stepped-wedge cluster randomized clinical trial was conducted across 20 weeks (from June 17 to November 1, 2019) at 9 medical oncology clinics (8 subspecialty oncology and 1 general oncology clinics) within a large academic health system in Pennsylvania. Clinicians at the 2 smallest subspecialty clinics were grouped together, resulting in 8 clinic groups randomly assigned to the 4 intervention wedge periods. Included participants in the intention-to-treat analyses were 78 oncology clinicians who received SIC training and their patients (N = 14 607) who had an outpatient oncology encounter during the study period.

    Interventions  (1) Weekly emails to oncology clinicians with SIC performance feedback and peer comparisons; (2) a list of up to 6 high-risk patients (≥10% predicted risk of 180-day mortality) scheduled for the next week, estimated using a validated machine learning algorithm; and (3) opt-out text message prompts to clinicians on the patient’s appointment day to consider an SIC. Clinicians in the control group received usual care consisting of weekly emails with cumulative SIC performance.

    Main Outcomes and Measures  Percentage of patient encounters with an SIC in the intervention group vs the usual care (control) group.

    Results  The sample consisted of 78 clinicians and 14 607 patients. The mean (SD) age of patients was 61.9 (14.2) years, 53.7% were female, and 70.4% were White. For all encounters, SICs were conducted among 1.3% in the control group and 4.6% in the intervention group, a significant difference (adjusted difference in percentage points, 3.3; 95% CI, 2.3-4.5; P < .001). Among 4124 high-risk patient encounters, SICs were conducted among 3.6% in the control group and 15.2% in the intervention group, a significant difference (adjusted difference in percentage points, 11.6; 95% CI, 8.2-12.5; P < .001).

    Conclusions and Relevance  In this stepped-wedge cluster randomized clinical trial, an intervention that delivered machine learning mortality predictions with behavioral nudges to oncology clinicians significantly increased the rate of SICs among all patients and among patients with high mortality risk who were targeted by the intervention. Behavioral nudges combined with machine learning mortality predictions can positively influence clinician behavior and may be applied more broadly to improve care near the end of life.

    Trial Registration  ClinicalTrials.gov Identifier: NCT03984773

    Introduction

    Patients with cancer often receive treatment and acute care use, particularly near the end of life, that does not align with their preferences or goals of care.1-5 Early discussions about goals and treatment preferences may lead to better perceived quality of life, reduced emotional distress, and decreased health care use near the end of life.6-8 However, most patients with cancer die without a documented discussion about goals and treatment preferences.9,10

    There are several reasons for the dearth of such discussions between oncologists and patients with cancer. First, oncologists correctly identify fewer than half of their patients who will die within 6 to 12 months.11,12 Overoptimistic assessments of short-term mortality risk may result in inadequate identification of patients who will benefit from a timely discussion.13-15 Second, it is often difficult to change clinician behavior and practice patterns. Insights from behavioral economics, including changing default options and delivering performance feedback, have been shown to alter clinician behavior in a variety of clinical settings.16-19

    Serious illness conversations (SICs) are a type of advance care planning discussion and are structured conversations between clinicians, patients, and families intended to explore prognostic awareness, treatment priorities, and goals of care.20 In a prior randomized clinical trial, an intervention that provided clinicians with a Serious Illness Conversation Guide ameliorated anxiety and depression among patients with cancer.21,22 The objective of the present stepped-wedge cluster randomized clinical trial was to test the effect of an intervention that integrated real-time machine learning (ML) mortality predictions with behavioral nudges on motivating SICs between oncologists and high-risk patients. We used a validated ML algorithm that has been shown to accurately predict 180-day mortality risk.23 We hypothesized that this multimodal intervention would increase the rate of SICs between oncology clinicians and all patients as well as between oncology clinicians and high-risk patients.

    Methods
    Study Design

    This was a stepped-wedge cluster randomized clinical trial (NCT03984773) among 9 medical oncology clinics in the University of Pennsylvania Health System.24 The trial included five 4-week wedges and was conducted from June 17 to November 1, 2019. The trial evaluated the effect of delivering ML mortality estimates with behavioral nudges on the rate of SICs among patients with cancer. The mortality prediction algorithm has been previously validated.23 This trial followed the Consolidated Standards of Reporting Trials (CONSORT) reporting guideline. The trial protocol was approved by the University of Pennsylvania institutional review board (trial protocol in Supplement 1),25 which also granted a waiver for the requirement to obtain informed consent because this study was an evaluation of a health system initiative that posed minimal risk to clinicians and patients.

    Study Sample

    Eligible clinicians were physicians, physician assistants, and nurse practitioners who provided oncology care at 8 subspecialty clinics (breast, central nervous system, gastrointestinal, genitourinary, lymphoma, melanoma, myeloma, and thoracic or head and neck) and 1 general oncology clinic (n = 78). Clinicians at the 2 smallest subspecialty clinics (melanoma and central nervous system malignant neoplasms) were grouped together, resulting in 8 clinic groups. Clinicians were excluded if they cared only for patients with benign hematologic or genetic disorders (n = 3) saw fewer than 12 patients classified as high risk by the algorithm in either the preintervention or postintervention period (n = 6), or had not undergone SIC training at the time of trial initiation (n = 4).

    Eligible patients had encounters at 1 of the clinics during the study period. Patients were excluded if they had a documented SIC or advanced care planning conversation (ACP) prior to the start of the trial, or if they were enrolled in another ongoing trial of early palliative care. Genetics encounters were also excluded.

    Randomization

    The 8 clinic groups were randomized to the 4 intervention wedge periods, stratified by SIC rate (above or below the median). The study senior author (M.S.P.) and data analyst (C.A.L.R.) were blinded to wedge assignment until analysis of the primary end point was complete. Blinding was not feasible for investigators who were oncologists practicing at Penn Medicine.

    Interventions

    All clinicians included in the trial were trained in the use of the Serious Illness Conversation Guide (Ariadne Labs) at least 3 months prior to the start of the trial (range, 5-18 months).20 The default advance care planning template for clinicians was based on this guide.

    For approximately 1 year before the trial began, clinicians in all clinic groups received weekly emails that reported their individual SIC performance and blinded peer comparison in 2 bar graphs: 1 for cumulative SICs performed since that clinician’s SIC training date and 1 for SICs performed in the prior week. This continued during the control period of the trial and was turned off once the intervention was implemented. In the first 4-week wedge, all clinic groups remained in usual care (control). In each subsequent 4-week period, 2 clinic groups (intervention) received the intervention. By the start of the fifth wedge, all clinic groups were receiving the intervention (eFigure 1 in the Supplement 2).

    The intervention was directed toward clinicians and included 3 main components. First, every Thursday, each clinician received an email with performance feedback indicating the number of SICs he or she performed in the prior 4 weeks. The email also contained a peer comparison message; this message differed for clinicians who performed below or above 8 SICs or who were among the top 10 performers of SICs during the prior 4 weeks. Samples of the email text are shown in eFigure 2 in the Supplement 2. Performance feedback and peer comparisons have been shown to be successful in motivating clinician decision-making.17,26,27

    Second, the email included a link to a secure dashboard in which clinicians could review patients scheduled for the following week in their clinic who had a high risk of mortality; these were described as patients “who may benefit from an SIC.” Mortality risk was determined by a previously validated ML algorithm, which used structured electronic health record (EHR) data to predict risk of 180-day mortality.2 Every Thursday morning, the algorithm generated mortality risk predictions for all patients with a scheduled encounter during the following week with an eligible clinician. Clinicians could view a personalized list of up to 6 patients with the highest predicted 180-day mortality risk (“high-risk” patients) via a secure dashboard. Patients with a mortality risk below 10% were not included in the list. Mortality predictions were updated weekly; thus, patient assignment as “high risk” could change from encounter to encounter. Information in the dashboard included patient identifiers, the author and date of any prior SICs, and a checkbox to opt out of reminder texts. Using opt out (as opposed to opt in) as a default has been successful in different health care settings to change physician behavior.28-32 The prediction algorithm silently identified high-risk patients during the control periods, but these lists were not shared with the clinicians via email or text.

    Third, clinicians received a text message reminding them to consider having an SIC with any high-risk patient. For the high-risk patients on the clinicians’ dashboard, clinicians were sent this message on the morning of the patient’s encounter, unless they opted out or an SIC was performed in the prior 2 months.

    Outcome Measures

    The primary outcome was change in SIC rate among all patient encounters. Secondary outcomes included changes in rates of (1) SICs among high-risk patients and (2) ACPs for the overall sample and for the high-risk subgroup.

    The electronic medical record had an ACP section in which advance care planning notes were documented. Any note documented in this section was classified as an ACP. The subset of notes that included a preformatted SIC template was classified as an SIC. Thus, ACPs encompassed both SIC and non-SIC documentation about treatment goals and wishes.

    Statistical Analysis

    A priori power calculations used data from the University of Pennsylvania Health System to estimate the rate of SICs. Assuming a baseline SIC rate of 0.65%, we estimated that this trial would have 80% power to detect a 60% relative increase in SIC rate using a 2-sided α of 0.05 as our threshold for statistical significance.

    All trial clinicians and their patients were included in the intention-to-treat analysis. Analyses were conducted using the first patient encounter in each wedge; subsequent encounters were excluded to avoid biasing results toward patients with numerous encounters. Patients seen by physician assistants and nurse practitioners were allocated to their corresponding oncologist. Because care planning documentation can occur days to weeks after an encounter, SICs and ACPs within 28 days of the encounter with no intervening encounter were counted in the primary outcome. We performed a sensitivity analysis that defined the primary outcome using a shorter, 7-day period after the index encounter to count SICs and ACPs.

    Similar to prior work,7 we fit models for the outcome measures based on generalized estimating equations with a logit link and an independence correlation structure using oncologist as the clustering unit. The main models used patient-encounter level observations, clinic-group and wedge-period fixed effects, and a binary indicator for intervention implementation. To test the robustness of our findings, we fit a fully adjusted model that also included age, sex, marital status, patient-reported race/ethnicity, insurance status, and Charlson Comorbidity Index. To obtain the adjusted difference and 95% CIs in the percentage points, we used the bootstrap method, resampling patients 1000 times.33,34 Resampling of patients was conducted by the oncologist variable to maintain clustering at the oncologist level. Analyses were conducted between January 6 and March 9, 2020. Two-sided hypothesis tests used a significance level of P ≤ .05. All analyses were conducted using SAS, version 9.4 (SAS Institute Inc).

    Results

    The sample consisted of 78 clinicians (of whom 42 were oncologists), 14 607 patients, and 26 059 patient encounters (Figure). Patients’ mean (SD) age was 61.9 (14.2) years. Of 14 607 patients, 8141 (55.7%) were female, 10 285 (70.4%) were non-Hispanic White, 6876 (47.1%) were commercially insured, 6806 (46.7%) were covered by Medicare, and 925 (6.3)% were covered by Medicaid (Table 1). Patients in the control and intervention groups had similar characteristics. High-risk encounters accounted for 2125 of 12 170 patient encounters (17.5%) in the control groups and 1999 of 13 889 patient encounters (14.4%) in the intervention groups. The median 180-day mortality risk per patient-encounter was similar in the control and intervention groups (4% [interquartile range {IQR}, 2%-13%] vs 4% [IQR, 2%-12%]). Among high-risk patients, the patients’ characteristics and median mortality risk (34% [IQR, 20%-53%] vs 36% [IQR, 20%-56%]) were also similar between control and intervention groups (eTable 1 in Supplement 2). The number of patient encounters and SIC rates by clinic and control vs intervention group are shown in eTable 2 in Supplement 2.

    Serious Illness Conversations

    Among all patients, SICs were conducted in 155 of 12 170 control encounters (1.3%) and 632 of 13 889 intervention encounters (4.6%). Among high-risk patients, SICs were conducted in 77 of 2125 control encounters (3.6%) and 304 of 1999 intervention encounters (15.2%) (eFigure 3 in Supplement 2). In adjusted analyses of all patients, the intervention led to a significant increase in the SIC rate (adjusted difference in percentage points, 3.3; 95% CI, 2.3-4.5; P < .001) (Table 2). In adjusted analyses of 4124 high-risk patient encounters, there was a significant increase in the SIC rate (adjusted difference in percentage points, 11.6; 95% CI, 8.2-12.5; P < .001). Full model results are shown in eTable 3 and eTable 4 in Supplement 2. Sensitivity analyses showed similar results (eTables 5, 6, 7, and 12 in Supplement 2).

    Advanced Care Planning Conversations

    Among all patients, ACPs were conducted in 231 of 12 170 control encounters (1.9%) and 680 of 13 889 intervention encounters (4.9%). Among high-risk patients, ACPs were conducted in 124 of 2125 control encounters (5.8%) and 350 of 1999 intervention encounters (17.5%). In adjusted analyses of all patients, the intervention led to a significant increase in the ACP rate (adjusted difference in percentage points, 3.0; 95% CI, 2.1-4.1; P = .001) (Table 2). In adjusted analyses of high-risk patients, the intervention led to a significant increase in the ACP rate (adjusted difference in percentage points, 11.7; 95% CI, 8.4-15.7; P < .001). Full model results are shown in eTable 8 and eTable 9 in Supplement 2. Sensitivity analyses showed similar results (eTables 10, 11 and 12 in Supplement 2). No adverse events were reported.

    Discussion

    In this randomized clinical trial conducted in 9 oncology practices, an intervention that combined ML mortality predictions with behavioral nudges significantly increased rates of SICs and ACPs. The intervention led to a 4-fold increase (3.6%-15.2%) in SICs among high-risk patients targeted by the intervention. To our knowledge, this is one of the first randomized clinical trials testing the effect of real-time ML predictions and behavioral nudges on clinical care. There are several important implications of our findings.

    First, our intervention resulted in a large increase in SIC rates, especially among patients at highest risk of 180-day mortality. Although prior studies have assessed SIC and ACP rates among decedents,6,7 our study showed that a behaviorally informed intervention led to a rapid increase in such conversations among all patients with cancer. Although the optimal SIC prevalence among patients with cancer is unknown, the effect of this intervention compares favorably with other published interventions on increasing the number of such conversations.35-37 Discussions about goals and treatment preferences have been associated with less aggressive care for cancer patients near the end of life, including increased hospice utilization rates and reduced chemotherapy and hospitalization rates near the end of life.1-8 Furthermore, documentation of such discussions ensures that patients’ wishes are recorded and available across care settings in a health system. Our study has particular importance as oncologists and patients now grapple with care delays and competing risks due to the coronavirus disease 2019 pandemic. Indeed, documentation of goals and wishes has been recognized by several oncology guideline bodies as a key priority to ensure goal-concordant care during the pandemic. A uniform approach to the automated identification of high-risk patients that increases the rate of SICs can be combined with new care models, such as telemedicine, to meet this need.

    Targeting SICs to high-risk patients at risk of short-term mortality may result in improved goal-concordant care, which was not shown in a previous randomized clinical trial of an intervention consisting primarily of training on the Serious Illness Conversation Guide.21,22 The cohort in that trial consisted of individuals who died. The use of that cohort may explain why overall rates of SICs in the control and intervention groups were much higher than those in the present trial, which consisted of all patients (decedents and nondecedents) with cancer, including many with curable early-stage cancers for whom SICs would not be expected. In addition, the relatively low SIC prevalence among even high-risk patients after the intervention may be due to the exclusion of patients with an existing SIC at the start of the present trial, a short follow-up period, clinician time constraints, the performance of SICs that were not documented in a trackable fashion in the EHR, and misclassification of patients as high risk. Subsequent research should determine longer-term clinical and quality-of-life outcomes associated with SICs prompted by this intervention.

    Second, although several trials have prospectively integrated ML risk predictions into clinical care, few studies have shown that interventions that involve ML predictions can positively influence clinical care or decision-making.38,39 Our intervention illustrates that generalizable behavioral principles combined with ML predictions may inform more accurate, individualized prognoses that clinicians and patients use to guide treatment decisions and ensure receipt of goal-concordant care. A key innovation of the present study was integrating behavioral economic principles into the ML-based intervention, including peer comparison, performance feedback, and opt-out reminders, which have been used successfully in other clinical settings to change clinician behavior.16-19,40 Future implementation of artificial intelligence or ML tools in clinical practice should explore similar mechanisms of delivering predictions and integrating behavioral principles to maximally improve clinician and patient decision-making. Yet these behavioral nudges may not be equally necessary and effective. Although the present study cannot disentangle the effects of individual intervention components, setting the default to a desired behavior—in this instance, defaulting clinicians to receive notifications encouraging SICs with high-risk patients—is a powerful nudge that has been widely applied in medicine to change behavior and may be the most impactful component.28 Future research should use head-to-head comparisons to identify which intervention components deliver the most benefit and how an integrated EHR-based mortality prediction tool can be generalized to other health systems.

    Third, although our intervention effect was strongest in the high-risk patients flagged by the algorithm, we still detected an increase of nearly 2 percentage points in SICs among patients who were not deemed high-risk by the algorithm. These “off-target” patients comprised the majority of our cohort. To our knowledge, off-target effects of an intervention to promote supportive care have not been shown. This off-target effect may have occurred because the clinicians who received the intervention became accustomed to actively considering mortality risk in their clinical practice, resulting in an increase in SICs for patients who may have a poor prognosis but were not flagged by the algorithm as high risk.

    Strengths and Limitations

    Our study has several strengths, including the application of behavioral insights into an ML-based intervention, integration of accurate individual-level mortality risk predictions at the point of care, a stepped-wedge design that controlled for cancer site–specific uptake of SICs, and a large sample of patients across several different cancer types and in both tertiary academic and general oncology practices. This study has several limitations. First, this study was performed in a single academic health system, and the participants may not be representative of the general population of oncologists or patients with cancer. However, given the prominence of EHRs in oncology care and the many examples of EHR-based ML algorithms across EHR types, the principles behind this intervention could theoretically be applied in any cancer care setting.41 Second, early ACPs and SICs are surrogates for goal-concordant care and decreased aggressive end-of-life care. Longer-term studies are needed to assess whether algorithm-based interventions improve clinical outcomes for patients. Third, our intervention consisted of several components, including a real-time prediction algorithm, a secure list of high-risk patients, peer comparisons, performance reports, and text messages. Future work is needed to assess the individual effects of each of these interventions.

    Conclusions

    An intervention combining ML mortality risk predictions and behavioral nudges led to an increase in SICs among patients with cancer. This study provides guidance for more robust integration of automated risk predictions alongside behavioral interventions in future prospective studies related to and outside of SICs in oncology.

    Back to top
    Article Information

    Accepted for Publication: July 28, 2020.

    Published Online: October 15, 2020. doi:10.1001/jamaoncol.2020.4759

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2020 Manz CR et al. JAMA Oncology.

    Corresponding Author: Christopher R. Manz, MD, Dana-Farber Cancer Institute, 450 Brookline Avenue, 11th Floor, Boston, MA 02215 (christopher_manz@dfci.harvard.edu).

    Author Contributions: Drs Manz and Parikh had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Drs Manz and Parikh contributed equally to this work and are co–first authors.

    Concept and design: Manz, Parikh, Small, Chivers, Regli, Hanson, O’Connor, Schuchter, Shulman, Patel.

    Acquisition, analysis, or interpretation of data: Manz, Parikh, Small, Evans, Chivers, Bekelman, Rareshide, O’Connor, Shulman, Patel.

    Drafting of the manuscript: Manz, Parikh, Evans, Bekelman.

    Critical revision of the manuscript for important intellectual content: Manz, Parikh, Small, Chivers, Regli, Hanson, Bekelman, Rareshide, O’Connor, Schuchter, Shulman, Patel.

    Statistical analysis: Manz, Parikh, Small, Chivers, Rareshide.

    Obtained funding: Manz, Parikh.

    Administrative, technical, or material support: Parikh, Evans, Chivers, Regli, O’Connor, Shulman, Patel.

    Supervision: Parikh, Regli, Hanson, O’Connor, Schuchter, Shulman, Patel.

    Conflict of Interest Disclosures: Dr Parikh reported receiving personal fees from GNS Healthcare and the Cancer Study Group and grants from Penn Center for Precision Medicine and the MUSC Transdisciplinary Collaborative Center in Precision Medicine and Minority Men’s Health, and grants and nonfinancial support from the Conquer Cancer Foundation outside the submitted work. Dr Bekelman reported receiving personal fees from the Centers for Medicare and Medicaid Services, Optum, CVS Health, the National Comprehensive Cancer Network, and UnitedHealthcare and grants from Pfizer, North Carolina Blue Cross Blue Shield, Embedded Healthcare, and UnitedHealth Group outside the submitted work. Dr Patel reported being an advisory board member of and receiving stock options from Life.io and personal fees from Catalyst Health LLC, HealthMine Services, and Holistic Industries outside the submitted work. No other disclosures were reported.

    Funding/Support: This study was funded by the Penn Center for Precision Medicine Accelerator Fund (to Drs Manz and Parikh), award T32 GM075766-14 to Dr Manz from the National Institute of General Medical Sciences, and the University of Pennsylvania Health System through the Penn Medicine Nudge Unit.

    Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Meeting Presentation: This paper was presented at the Annual Meeting of the American Society of Clinical Oncology; May 29, 2020; virtual; and at the Annual Research Meeting of the Academy Health; July 28, 2020; virtual.

    Data Sharing Statement: See Supplement 3.

    Additional Contributions: Peter E. Gabriel, MD, MSE, Penn Medicine, provided assistance with database curation; and Jennifer Braun, MHA, BSN, Penn Medicine, provided guidance on interpretation. Neither person receive financial compensation for their contributions.

    References
    1.
    Emanuel  EJ, Young-Xu  Y, Levinsky  NG, Gazelle  G, Saynina  O, Ash  AS.  Chemotherapy use among Medicare beneficiaries at the end of life.   Ann Intern Med. 2003;138(8):639-643. doi:10.7326/0003-4819-138-8-200304150-00011 PubMedGoogle ScholarCrossref
    2.
    Earle  CC, Neville  BA, Landrum  MB, Ayanian  JZ, Block  SD, Weeks  JC.  Trends in the aggressiveness of cancer care near the end of life.   J Clin Oncol. 2004;22(2):315-321. doi:10.1200/JCO.2004.08.136 PubMedGoogle ScholarCrossref
    3.
    Earle  CC, Landrum  MB, Souza  JM, Neville  BA, Weeks  JC, Ayanian  JZ.  Aggressiveness of cancer care near the end of life: is it a quality-of-care issue?   J Clin Oncol. 2008;26(23):3860-3866. doi:10.1200/JCO.2007.15.8253 PubMedGoogle ScholarCrossref
    4.
    Chastek  B, Harley  C, Kallich  J, Newcomer  L, Paoli  CJ, Teitelbaum  AH.  Health care costs for patients with cancer at the end of life.   J Oncol Pract. 2012;8(6):75s-80s. doi:10.1200/JOP.2011.000469 PubMedGoogle ScholarCrossref
    5.
    Wen  F-H, Chen  J-S, Su  P-J,  et al.  Terminally ill cancer patients’ concordance between preferred life-sustaining treatment states in their last six months of life and received life-sustaining treatment states in their last month: an observational study.   J Pain Symptom Manage. 2018;56(4):509-518.e3. doi:10.1016/j.jpainsymman.2018.07.003 PubMedGoogle ScholarCrossref
    6.
    Bernacki  RE, Block  SD; American College of Physicians High Value Care Task Force.  Communication about serious illness care goals: a review and synthesis of best practices.   JAMA Intern Med. 2014;174(12):1994-2003. doi:10.1001/jamainternmed.2014.5271 PubMedGoogle ScholarCrossref
    7.
    Wright  AA, Zhang  B, Ray  A,  et al.  Associations between end-of-life discussions, patient mental health, medical care near death, and caregiver bereavement adjustment.   JAMA. 2008;300(14):1665-1673. doi:10.1001/jama.300.14.1665 PubMedGoogle ScholarCrossref
    8.
    Brinkman-Stoppelenburg  A, Rietjens  JA, van der Heide  A.  The effects of advance care planning on end-of-life care: a systematic review.   Palliat Med. 2014;28(8):1000-1025. doi:10.1177/0269216314526272 PubMedGoogle ScholarCrossref
    9.
    Schubart  JR, Levi  BH, Bain  MM, Farace  E, Green  MJ.  Advance care planning among patients with advanced cancer.   J Oncol Pract. 2019;15(1):e65-e73. doi:10.1200/JOP.18.00044 PubMedGoogle ScholarCrossref
    10.
    National Quality Forum: Palliative and end-of-life care 2015-2016. Draft report for comment June 20, 2016. Accessed June 10, 2020. http://www.qualityforum.org/Projects/n-r/Palliative_and_End-of-Life_Care_Project_2015-2016/Draft_Report_for_Comment.aspx
    11.
    Chow  E, Harth  T, Hruby  G, Finkelstein  J, Wu  J, Danjoux  C.  How accurate are physicians’ clinical predictions of survival and the available prognostic tools in estimating survival times in terminally ill cancer patients? a systematic review.   Clin Oncol (R Coll Radiol). 2001;13(3):209-218. doi:10.1007/s001740170078 PubMedGoogle Scholar
    12.
    Ermacora  P, Mazzer  M, Isola  M,  et al.  Prognostic evaluation in palliative care: final results from a prospective cohort study.   Support Care Cancer. 2019;27(6):2095-2102. doi:10.1007/s00520-018-4463-z PubMedGoogle ScholarCrossref
    13.
    Weeks  JC, Cook  EF, O’Day  SJ,  et al.  Relationship between cancer patients’ predictions of prognosis and their treatment preferences.   JAMA. 1998;279(21):1709-1714. doi:10.1001/jama.279.21.1709 PubMedGoogle ScholarCrossref
    14.
    Rose  JH, O’Toole  EE, Dawson  NV,  et al.  Perspectives, preferences, care practices, and outcomes among older and middle-aged patients with late-stage cancer.   J Clin Oncol. 2004;22(24):4907-4917. doi:10.1200/JCO.2004.06.050 PubMedGoogle ScholarCrossref
    15.
    White  N, Reid  F, Harris  A, Harries  P, Stone  P.  A systematic review of predictions of survival in palliative care: how accurate are clinicians and who are the experts?   PLoS One. 2016;11(8):e0161407. doi:10.1371/journal.pone.0161407 PubMedGoogle Scholar
    16.
    Brown  B, Gude  WT, Blakeman  T,  et al.  Clinical Performance Feedback Intervention Theory (CP-FIT): a new theory for designing, implementing, and evaluating feedback in health care based on a systematic review and meta-synthesis of qualitative research.   Implement Sci. 2019;14(1):40. doi:10.1186/s13012-019-0883-5 PubMedGoogle ScholarCrossref
    17.
    Navathe  AS, Emanuel  EJ.  Physician peer comparisons as a nonfinancial strategy to improve the value of care.   JAMA. 2016;316(17):1759-1760. doi:10.1001/jama.2016.13739 PubMedGoogle ScholarCrossref
    18.
    Patel  MS, Volpp  KG, Asch  DA.  Nudge units to improve the delivery of health care.   N Engl J Med. 2018;378(3):214-216. doi:10.1056/NEJMp1712984 PubMedGoogle ScholarCrossref
    19.
    Patel  MS, Volpp  KG, Small  DS,  et al.  Using active choice within the electronic health record to increase influenza vaccination rates.   J Gen Intern Med. 2017;32(7):790-795. doi:10.1007/s11606-017-4046-6 PubMedGoogle ScholarCrossref
    20.
    Bernacki  R, Hutchings  M, Vick  J,  et al.  Development of the Serious Illness Care Program: a randomised controlled trial of a palliative care communication intervention.   BMJ Open. 2015;5(10):e009032. doi:10.1136/bmjopen-2015-009032 PubMedGoogle Scholar
    21.
    Paladino  J, Bernacki  R, Neville  BA,  et al.  Evaluating an intervention to improve communication between oncology clinicians and patients with life-limiting cancer: a cluster randomized clinical trial of the Serious Illness Care Program.   JAMA Oncol. 2019;5(6):801-809. doi:10.1001/jamaoncol.2019.0292 PubMedGoogle ScholarCrossref
    22.
    Bernacki  R, Paladino  J, Neville  BA,  et al.  Effect of the Serious Illness Care Program in outpatient oncology: a cluster randomized clinical trial.   JAMA Intern Med. 2019;179(6):751-759. doi:10.1001/jamainternmed.2019.0077 PubMedGoogle ScholarCrossref
    23.
    Parikh  RB, Manz  C, Chivers  C,  et al.  Machine learning approaches to predict 6-month mortality among patients with cancer.   JAMA Netw Open. 2019;2(10):e1915997. doi:10.1001/jamanetworkopen.2019.15997 PubMedGoogle Scholar
    24.
    Machine-generated mortality estimates and nudges to promote advance care planning discussion among cancer patients. ClinicalTrials.gov identifier: NCT03984773. Accessed September 2, 2020. https://clinicaltrials.gov/ct2/show/NCT03984773
    25.
    Manz  CR, Parikh  RB, Evans  CN,  et al.  Integrating machine-generated mortality estimates and behavioral nudges to promote serious illness conversations for cancer patients: design and methods for a stepped-wedge cluster randomized controlled trial.   Contemp Clin Trials. 2020;90:105951. doi:10.1016/j.cct.2020.105951 PubMedGoogle Scholar
    26.
    Patel  MS, Kurtzman  GW, Kannan  S,  et al.  Effect of an automated patient dashboard using active choice and peer comparison performance feedback to physicians on statin prescribing: the prescribe cluster randomized clinical trial.   JAMA Netw Open. 2018;1(3):e180818. doi:10.1001/jamanetworkopen.2018.0818 PubMedGoogle Scholar
    27.
    Meeker  D, Linder  JA, Fox  CR,  et al.  Effect of behavioral interventions on inappropriate antibiotic prescribing among primary care practices: a randomized clinical trial.   JAMA. 2016;315(6):562-570. doi:10.1001/jama.2016.0275 PubMedGoogle ScholarCrossref
    28.
    Halpern  SD, Ubel  PA, Asch  DA.  Harnessing the power of default options to improve health care.   N Engl J Med. 2007;357(13):1340-1344. doi:10.1056/NEJMsb071595PubMedGoogle ScholarCrossref
    29.
    Patel  MS, Day  SC, Halpern  SD,  et al.  Generic medication prescription rates after health system–wide redesign of default options within the electronic health record.   JAMA Intern Med. 2016;176(6):847-848. doi:10.1001/jamainternmed.2016.1691 PubMedGoogle ScholarCrossref
    30.
    Patel  MS, Day  S, Small  DS,  et al.  Using default options within the electronic health record to increase the prescribing of generic-equivalent medications: a quasi-experimental study.   Ann Intern Med. 2014;161(10)(suppl):S44-S52. doi:10.7326/M13-3001PubMedGoogle ScholarCrossref
    31.
    Sharma  S, Guttmann  D, Small  DS,  et al.  Effect of introducing a default order in the electronic medical record on unnecessary daily imaging during palliative radiotherapy for adults with cancer: a stepped-wedge cluster randomized clinical trial.   JAMA Oncol. 2019;5(8):1220. doi:10.1001/jamaoncol.2019.1432 PubMedGoogle ScholarCrossref
    32.
    Delgado  MK, Shofer  FS, Patel  MS,  et al.  Association between electronic medical record implementation of default opioid prescription quantities and prescribing behavior in two emergency departments.   J Gen Intern Med. 2018;33(4):409-411. doi:10.1007/s11606-017-4286-5 PubMedGoogle ScholarCrossref
    33.
    Efron  B, Tibshirani  R.  An Introduction to the Bootstrap. New York, NY: Chapman & Hall; 1993. doi:10.1007/978-1-4899-4541-9
    34.
    Davison  AC, Hinkley  DV.  Bootstrap Methods and Their Application. New York, NY: Cambridge University Press; 1997. doi:10.1017/CBO9780511802843
    35.
    Schofield  G, Kreeger  L, Meyer  M,  et al.  Implementation of a quality improvement programme to support advance care planning in five hospitals across a health region.   BMJ Support Palliat Care. 2015;5(1):91-94. doi:10.1136/bmjspcare-2013-000574 PubMedGoogle ScholarCrossref
    36.
    Temel  JS, Greer  JA, Gallagher  ER,  et al.  Electronic prompt to improve outpatient code status documentation for patients with advanced lung cancer.   J Clin Oncol. 2013;31(6):710-715. doi:10.1200/JCO.2012.43.2203 PubMedGoogle ScholarCrossref
    37.
    Bestvina  CM, Polite  BN.  Implementation of advance care planning in oncology: a review of the literature.   J Oncol Pract. 2017;13(10):657-662. doi:10.1200/JOP.2017.021246 PubMedGoogle ScholarCrossref
    38.
    Titano  JJ, Badgeley  M, Schefflein  J,  et al.  Automated deep-neural-network surveillance of cranial images for acute neurologic events.   Nat Med. 2018;24(9):1337-1341. doi:10.1038/s41591-018-0147-y PubMedGoogle ScholarCrossref
    39.
    Lin  H, Li  R, Liu  Z,  et al.  Diagnostic efficacy and therapeutic decision-making capacity of an artificial intelligence platform for childhood cataracts in eye clinics: a multicentre randomized controlled trial.   EClinicalMedicine. 2019;9:52-59. doi:10.1016/j.eclinm.2019.03.001 PubMedGoogle ScholarCrossref
    40.
    Emanuel  EJ, Ubel  PA, Kessler  JB,  et al.  Using behavioral economics to design physician incentives that deliver high-value care.   Ann Intern Med. 2016;164(2):114-119. doi:10.7326/M15-1330 PubMedGoogle ScholarCrossref
    41.
    LeBlanc  TW, Back  AL, Danis  M, Abernethy  AP.  Electronic health records (EHRs) in the oncology clinic: how clinician interaction with EHRs can improve communication with the patient.   J Oncol Pract. 2014;10(5):317-321. doi:10.1200/JOP.2014.001385 PubMedGoogle ScholarCrossref
    ×