Long-term Effect of Machine Learning–Triggered Behavioral Nudges on Serious Illness Conversations and End-of-Life Outcomes Among Patients With Cancer

Key Points Question Does an intervention consisting of machine learning mortality predictions and behavioral nudges to oncology clinicians have an impact on serious illness conversations (SICs) and end-of-life outcomes among patients with cancer? Findings In this randomized clinical trial that included 20 506 patients with cancer (for a total of 41 021 encounters), the intervention led to a significant increase in SICs, from 3.4% to 13.5%, among encounters with patients at high risk of death while decreasing rates of end-of-life systemic therapy from 10.4% to 7.5%. Meaning These results suggest that a machine learning–based behavioral intervention can lead to an increase in SICs and reduction in end-of-life systemic therapy among outpatients with cancer.

11.5 Data safety and monitoring 36 11.6 Risk/benefit 37 11.6.1 Potential study risks 38 11.6.2 Potential study benefits 39 1. Abstract 42 Patients with cancer often undergo costly therapy and acute care utilization that is discordant 43 with their wishes, particularly at the end of life. Early serious illness conversations (SIC) 44 improve goal-concordant care, and accurate prognostication is critical to inform the timing and 45 content of these discussions. In this project, we will evaluate a health system initiative that uses a 46 machine learning algorithm to predict patients with a higher risk of short-term mortality and then 47 prompts oncologists to SICs with these patients. In partnership with the health system, this will 48 be conducted as a cluster-randomized trial to evaluate its effect. 49 2. Overall objectives 50 The objective of the study is to evaluate the effect of a health system initiative using machine 51 learning algorithms and behavioral nudges to prompt oncologists to have serious illness 52 conversations with patients at high-risk of short-term mortality.     This study will use a stepped-wedge cluster randomized trial to evaluate a health system 129 initiative. Oncology practices will be randomly assigned in sequential four-week blocks to 130 receive the email prompt intervention, in which individual oncologists will receive an automated 131 weekly email detailing 1) how many serious illness conversations they have had, 2) how their 132 number of serious illness conversations compares to peer oncology providers across UPHS, and 133 3) a weekly roster of their upcoming patients at high risk of short-term mortality as determined 134 by our mortality prediction algorithm (see below), viewable on a HIPAA-compliant secure web 135 interface. Clinicians will receive a HIPAA compliant text message on the morning of the 136 appointment reminding them to consider a serious illness conversation with patients on the list.

137
Providers may opt out of this reminder on the web interface containing the weekly patient roster 138 of high risk patients. Prior to receiving the intervention, practices will receive current standard 139 communications regarding serious illness performance until they are randomized to the 140 intervention. Practices will be cluster-randomized to the intervention over a 16-week period, 141 after which all practice physicians will receive the email intervention.

142
Based on clinician and health system feedback, the intervention will be modified to remove the 143 peer comparison message. This will occur 12 weeks into the follow-up period.

144
Following the intervention, brief REDCap questionnaires will be sent to all clinicians who 145 participated in the trial to explore perceptions of the study intervention. This survey should take 146 no longer than five minutes for clinicians to complete and will assess their feedback on the 147 overall intervention in addition to specific components including automated identification of 148 patients, receiving text and email notifications, and identifying the appropriate patients. The 149 survey will also collect basic demographic information (age, gender, practice site, and comfort 150 with SICs).

151
Additionally, we plan to contact clinicians who participated in the intervention and invite them to components (emails and text messages), and additional strategies that could be implemented to improve rates of SICs with patients. In summary, we will use semi-structured interviews to elicit 159 clinicians' perspectives on the Conversation Connect intervention beyond the brief REDCap 160 surveys. Additionally, clinicians will be asked basic demographic questions regarding their age, 161 race, gender, practice setting, and number of years in practice. We will pilot this guide with 3-5 162 oncology clinicians and anticipate it will take approximately 20 minutes to complete. The guide 163 may be revised to better clarify questions or adjust timing during the initial first five subject 164 interviews. Clinicians will be interviewed at a mutually convenient time by phone. Each 165 interview will be audio-recorded with permission from the provider and subsequently transcribed 166 for analysis. All research personnel will have completed human subject protection modules prior 167 to initiating the study. The MMRL will conduct all of the interviews. The MMRL will review 168 transcripts of the interviews and iteratively develop the code book and analyze the content of the 169 interviews.

170
Additionally, we will plan on studying the impact of the COVID-19 pandemic on the output and 171 accuracy of our predictive model, and identify subpopulations for whom utilization declines 172 during COVID have led to the most decrements in the predictive accuracy of the Conversation 173 Connect predictive model. We will specifically assess heterogeneity by racial/ethnic groups 174 (White, Black, Hispanic, Asian, and Native American), insurance status (Medicare, Medicaid, 175 Commercial), low-income zip code, and area-level socioeconomic metrics (e.g. area-level 176 income level), in addition to by cancer type. We will also assess the impact of COVID-related

Study duration 180
The study is expected to begin in June 2019 and take 10 months (16 weeks for intervention + 24 181 weeks followup) to complete. The REDCap questionnaires are expected to be completed by 182 March 2021. The semi-structured interviews will take place from December 2020 to June 2021.

183
Our analysis of COVID-related impacts will take place from February 2021 to July 2021.   190 Patients will accrue to the trial as their clinical practice receives the email intervention. Eight

191
University of Pennsylvania oncology practices will be randomly assigned to one of four start 192 dates separated by four weeks, resulting in four pairs of clinics starting the intervention two 193 clinics at a time every four weeks over sixteen weeks. When a clinic reaches the assigned start date for the intervention arm, the clinicians will begin to receive the weekly email intervention 195 and text reminders. Based on previous studies and assuming a baseline SIC rate of 0.65 SICs per 196 provider per 4-weeks, we believe we will have over 80% power to detect a 60% increase in SIC 197 rates per provider per 4-weeks.

Key inclusion criteria 199
Oncologists must meet the following criteria to be eligible for the study: Information on oncology practices and their clinicians at the University of Pennsylvania Health 225 system will be identified by department leadership. High-risk patients will be identified by 226 applying our mortality prediction algorithm (which uses electronic health record data from 227 Clarity, an EPIC reporting database) to weekly oncology clinic schedules.

Subject compensation
No compensation will be offered in the intervention or REDCap survey. We will offer clinicians 230 who participate in the semi-structured interviews a compensation of a $30 gift card. A waiver of informed consent is requested. This is a health system initiative that will be 234 implemented. The study is to evaluate that initiative. Therefore, physicians and their patients 235 will not be consented as this is the standard of practice per the health system initiative. Without 236 a waiver of the consent, the initiative would still be implemented by the health system, but the 237 study would be infeasible. There are several additional reasons why we feel a waiver of consent 238 should be granted. First, it is not feasible to consent every physician and as mentioned this in a medical journal. Clarity will be used to identify documentation of SICs and ACP.

268
After identifying eligible oncologists, block randomization will occur at the clinic level (noting 269 that PCAM melanoma and CNS Oncology will be randomized together as both clinics have a 270 low number of providers). We will obtain baseline measures and plan to stratify the 271 randomization by those above and below median level of SICPs in March through May of 2019. All analyses will be conducted using intention-to-treat using the patient as the unit of analysis 274 and clustering at the level of the oncologist. Advanced practice providers (APPs) will receive the 275 intervention, but will be associated with the oncologist with whom they work for the purposes of 276 the analysis. All hypothesis tests will use a two-sided alpha of 0.05 as our threshold for statistical 277 significance.

278
The primary and secondary outcome measures will use a binary indicator representing the 279 presences of an SIC or ACP for each patient. The primary outcome will be expressed as a 280 standardized rate of documented SIC discussions (number of documented SIC notes / 100 unique 281 patient visits). In the main adjusted analysis, we will fit models using generalized estimating 282 equations cluster on oncologists, using group (oncology practices) and period (4-week 283 increments) fixed effects and adjusting for monthly temporal trends.

284
To test the robustness of our findings, we will perform sensitivity analyses that adjusts for 285 available patient characteristics and comorbidities such as demographics and the Charlson 286 Comorbidity Index.

287
Additional sensitivity analyses will include: 288 -Including patients enrolled in aforementioned palliative care lung cancer trial 289 -Analyzing results clustering at the level of the clinician (oncologist or APP) 290 We will use descriptive statistics to analyze responses from our REDCap surveys.

291
The semi-structured interviews will be transcribed and uploaded to NVivo12 Plus, a data 292 management software. We will use a modified content analysis approach with constant 293 comparative coding to analyze the interview transcripts. We will iteratively develop a codebook 294 following the initial interviews based on the structure of the interview guide. One of two 295 reviewers will code each transcript, and approximately 20-25% of total transcripts will be coded 296 by both reviewers to establish inter-reviewer reliability. The adequacy of the codebook will be 297 periodically assessed by the reviewers in partnership with the research team and modifications 298 will be made as their understanding of the data and emergent themes evolves over time. The 299 reviewers will meet regularly to discuss discrepancies and update the code book as needed, with 300 record keeping adequate to track changes to the code book and the rationale. Kappa statistics will be generated to estimate inter-rater reliability. Coding will then be reviewed to summarize key 302 themes, and representative quotes will be selected to illustrate those themes.

303
To assess the impact of the COVID-19 pandemic on rates of conversations, we will use a quasi-304 experimental interrupted time series analysis, using March 23 rd , 2020 as the date of the COVID 305 exposure, to assess whether the decline in overall risk scores and number of patients flagged as 306 high risk is significantly lower during the COVID pandemic. We will use descriptive statistics to 307 compare the distribution of demographic characteristics of high-risk patients (Conversation 308 Connect score >10%) before and after the pandemic. We will use an interrupted time series 309 analysis with historic control to assess the impact of the COVID pandemic on the accuracy of the Human Factors Data on physicians and patients will be obtained from Epic, Penn Data Store and Tableau. Any 344 information that is obtained will be used only for research purposes and to inform the behavioral 345 nudges described above. Information on individual patients will only be disclosed within the 346 study team. All study staff will be reminded of the confidential nature of the data collected and 347 contained in these databases.

348
Data regarding provider performance of Serious Illness Conversations are already shared among 349 providers and will continue to be shared in unblinded fashion as part of the trial. Data regarding 350 acute care utilization in the last 30 days for a provider's deceased patient panel will be shared 351 amongst providers as well. This will occur as part of the intervention but is planned to occur 352 occur regardless of trial approval as part of quality improvement efforts.

353
Data will be stored, managed, and analyzed on a secure, encrypted server behind the University Unit related to physician and patient behavior at UPHS. All study personnel that will use this 358 data are listed on the IRB application and have completed training in HIPAA standards and the 359 CITI human subjects research. Data access will be password protected. Whenever possible, data 360 will be deidentified for analysis.

Subject privacy 362
All efforts will be made by study staff to ensure subject privacy. Data will be evaluated in a de-363 identified manner whenever possible. We will require time and date of appointment and zip code 364 data of trial participants to define our exposure period and link to area-level socioeconomic data 365 from the American Community Survey. Information on physicians and patients will not be disclosed to anyone outside of the study team, 368 with the exception of provider level data (SIC rates, acute care utilization) that are deliberately 369 shared as a part of the behavioral nudges. The investigators will provide oversight for the study evaluation of this health system initiative.

372
Providers will use their clinical judgment to determine the appropriateness of initiating ACPs 373 with patients, in accordance with standard of care. 374 11.6 Risk/benefit 375 11.6.1 Potential study risks 376 The potential risks associated with this study are minimal. Breach of data is a potential risk that 377 will be mitigated by using HIPAA compliant and secure data platforms for the nudge 378 interventions (name of list platform and platform used to share info w/ MAs) and evaluation 379 (Nudge Unit server). As noted above, substantial data demonstrates that ACPs improve patient 380 goal-concordant care without any identified harms (despite concerns that ACPs may increase 381 psychosocial distress, the opposite has been found), so the negative impact on patients is 382 minimal.

383
The provider data that will be shared with providers is already shared in one form (in the case of 384 SIC rates) and is planned to be shared with providers in the near future independent of this trial 385 (in the case of acute care utilization near the end of life), so the trial does not exposure providers 386 to additional risk. 387 11.6.2 Potential study benefits 388 As described in the literature, patients may have improved quality of life and better goal-389 concordant care when exposed to ACPs, especially earlier in their disease course. An 390 intervention that prompts providers to have an ACP with patients at a high risk of death in the 391 next six months may increase the likelihood that these conversations occur and that they occur 392 earlier in the disease course. However, it is possible that patients will receive no benefit from this 393 study. 394 11.6.3 Risk/benefit assessment 395 The risk/benefit ratio is highly favorable given the potential benefit from eligible patients having 396 an SIC or ACP discussion with their provider and benefitting from better goal-concordant care 397 and that efforts have been put into place to minimize the risk of breach of data.