Effect of Integrating Machine Learning Mortality Estimates With Behavioral Nudges to Clinicians on Serious Illness Conversations Among Patients With Cancer

Key Points Question What is the effect of delivering machine learning mortality predictions with behavioral nudges to oncology clinicians on the rate of serious illness conversations with patients with cancer? Findings In this stepped-wedge cluster randomized clinical trial that included 14 607 patients with cancer, the intervention led to a significant increase in serious illness conversations from approximately 1% to 5% of all patient encounters and from approximately 4% to 15% of encounters with patients having high predicted mortality risk. Meaning Machine learning mortality predictions combined with behavioral nudges to clinicians led to an increased rate of serious illness conversations for patients with cancer.

11.5 Data safety and monitoring 36 11.6 Risk/benefit 37 11.6.1 Potential study risks 38 11.6.2 Potential study benefits 39 1. Abstract 42 Patients with cancer often undergo costly therapy and acute care utilization that is discordant 43 with their wishes, particularly at the end of life. Early serious illness conversations (SIC) 44 improve goal-concordant care, and accurate prognostication is critical to inform the timing and 45 content of these discussions. In this project, we will evaluate a health system initiative that uses a 46 machine learning algorithm to predict patients with a higher risk of short-term mortality and then 47 prompts oncologists to SICs with these patients. In partnership with the health system, this will 48 be conducted as a cluster-randomized trial to evaluate its effect. 49 2. Overall objectives 50 The objective of the study is to evaluate the effect of a health system initiative using machine 51 learning algorithms and behavioral nudges to prompt oncologists to have serious illness 52 conversations with patients at high-risk of short-term mortality.

68
Patients with cancer often undergo costly therapy and acute care utilization that is discordant 69 with their wishes, particularly at the end of life. SICs until late in the disease course and resulting in aggressive care near the end of life. 10,11 Existing prognostic aids in oncology are rarely used because they do not apply to most cancers 12, 76 13 , do not identify most patients who will die within 1 year 14 , and require time-consuming data 77 input 15 . Electronic health record (EHR)-based predictive algorithms can improve clinician 78 decision-making in acute care settings 16-18 , but it is unclear whether such algorithms can guide 79 clinicians to perform SICs. As oncologists strive to assess patients' goals earlier in the disease 80 course, accurate prognostication is critical to inform the timing and content of these discussions.

81
--References-- This study will use a stepped-wedge cluster randomized trial to evaluate a health system 129 initiative. Oncology practices will be randomly assigned in sequential four-week blocks to 130 receive the email prompt intervention, in which individual oncologists will receive an automated 131 weekly email detailing 1) how many serious illness conversations they have had, 2) how their 132 number of serious illness conversations compares to peer oncology providers across UPHS, and 133 3) a weekly roster of their upcoming patients at high risk of short-term mortality as determined 134 by our mortality prediction algorithm (see below), viewable on a HIPAA-compliant secure web 135 interface. Clinicians will receive a HIPAA compliant text message on the morning of the 136 appointment reminding them to consider a serious illness conversation with patients on the list.

137
Providers may opt out of this reminder on the web interface containing the weekly patient roster 138 of high risk patients. Prior to receiving the intervention, practices will receive current standard 139 communications regarding serious illness performance until they are randomized to the 140 intervention. Practices will be cluster-randomized to the intervention over a 16-week period, 141 after which all practice physicians will receive the email intervention.

Study duration 143
The study is expected to begin in June 2019 and take 10 months (16 weeks for intervention + 24 144 weeks followup) to complete.

151
Patients will accrue to the trial as their clinical practice receives the email intervention. Eight 152 University of Pennsylvania oncology practices will be randomly assigned to one of four start 153 dates separated by four weeks, resulting in four pairs of clinics starting the intervention two 154 clinics at a time every four weeks over sixteen weeks. When a clinic reaches the assigned start 155 date for the intervention arm, the clinicians will begin to receive the weekly email intervention 156 and text reminders. Based on previous studies and assuming a baseline SIC rate of 0.65 SICs per provider per 4-weeks, we believe we will have over 80% power to detect a 60% increase in SIC 158 rates per provider per 4-weeks. 160 Oncologists must meet the following criteria to be eligible for the study: Information on oncology practices and their clinicians at the University of Pennsylvania Health 186 system will be identified by department leadership. High-risk patients will be identified by 187 applying our mortality prediction algorithm (which uses electronic health record data from 188 Clarity, an EPIC reporting database) to weekly oncology clinic schedules. No compensation will be offered in this study.

193
A waiver of informed consent is requested. This is a health system initiative that will be 194 implemented. The study is to evaluate that initiative. Therefore, physicians and their patients 195 will not be consented as this is the standard of practice per the health system initiative. Without  After identifying eligible oncologists, block randomization will occur at the clinic level (noting 224 that PCAM melanoma and CNS Oncology will be randomized together as both clinics have a 225 low number of providers). We will obtain baseline measures and plan to stratify the 226 randomization by those above and below median level of SICPs in March through May of 2019.

Analysis plan
All analyses will be conducted using intention-to-treat using the patient as the unit of analysis 229 and clustering at the level of the oncologist. Advanced practice providers (APPs) will receive the 230 intervention, but will be associated with the oncologist with whom they work for the purposes of 231 the analysis. All hypothesis tests will use a two-sided alpha of 0.05 as our threshold for statistical 232 significance.

233
The primary and secondary outcome measures will use a binary indicator representing the 234 presences of an SIC or ACP for each patient. The primary outcome will be expressed as a 235 standardized rate of documented SIC discussions (number of documented SIC notes / 100 unique 236 patient visits). In the main adjusted analysis, we will fit models using generalized estimating 237 equations cluster on oncologists, using group (oncology practices) and period (4-week 238 increments) fixed effects and adjusting for monthly temporal trends.

239
To test the robustness of our findings, we will perform sensitivity analyses that adjusts for Data on physicians and patients will be obtained from Epic, Penn Data Store and Tableau. Any 275 information that is obtained will be used only for research purposes and to inform the behavioral 276 nudges described above. Information on individual patients will only be disclosed within the 277 study team. All study staff will be reminded of the confidential nature of the data collected and 278 contained in these databases.

279
Data regarding provider performance of Serious Illness Conversations are already shared among 280 providers and will continue to be shared in unblinded fashion as part of the trial. Data regarding 281 acute care utilization in the last 30 days for a provider's deceased patient panel will be shared 282 amongst providers as well. This will occur as part of the intervention but is planned to occur 283 occur regardless of trial approval as part of quality improvement efforts.

284
Data will be stored, managed, and analyzed on a secure, encrypted server behind the University Unit related to physician and patient behavior at UPHS. All study personnel that will use this 289 data are listed on the IRB application and have completed training in HIPAA standards and the 290 CITI human subjects research. Data access will be password protected. Whenever possible, data 291 will be deidentified for analysis.

Subject privacy 293
All efforts will be made by study staff to ensure subject privacy. Data will be evaluated in a de-294 identified manner whenever possible.

295
11.4 Data disclosure 296 Information on physicians and patients will not be disclosed to anyone outside of the study team, 297 with the exception of provider level data (SIC rates, acute care utilization) that are deliberately 298 shared as a part of the behavioral nudges.

Data safety and monitoring 300
The investigators will provide oversight for the study evaluation of this health system initiative. The potential risks associated with this study are minimal. Breach of data is a potential risk that 306 will be mitigated by using HIPAA compliant and secure data platforms for the nudge 307 interventions (name of list platform and platform used to share info w/ MAs) and evaluation 308 (Nudge Unit server). As noted above, substantial data demonstrates that ACPs improve patient 309 goal-concordant care without any identified harms (despite concerns that ACPs may increase 310 psychosocial distress, the opposite has been found), so the negative impact on patients is 311 minimal.

312
The provider data that will be shared with providers is already shared in one form (in the case of 313 SIC rates) and is planned to be shared with providers in the near future independent of this trial 314 (in the case of acute care utilization near the end of life), so the trial does not exposure providers 315 to additional risk. 316 11.6.2 Potential study benefits 317 As described in the literature, patients may have improved quality of life and better goal-318 concordant care when exposed to ACPs, especially earlier in their disease course. An 319 intervention that prompts providers to have an ACP with patients at a high risk of death in the 320 next six months may increase the likelihood that these conversations occur and that they occur 321 earlier in the disease course. However, it is possible that patients will receive no benefit from this 322 study.
323 11.6.3 Risk/benefit assessment 324 The risk/benefit ratio is highly favorable given the potential benefit from eligible patients having 325 an SIC or ACP discussion with their provider and benefitting from better goal-concordant care 326 and that efforts have been put into place to minimize the risk of breach of data.