Evaluation of Technology-Enabled Monitoring of Patient-Reported Outcomes to Detect and Treat Toxic Effects Linked to Immune Checkpoint Inhibitors

Key Points Question Can a technology-enabled, dynamically adaptive protocol be efficiently used to provide early and accurate detection of toxic effects to immune checkpoint inhibitors? Findings In this cohort study including 47 individuals with genitourinary cancers, a median patient adherence rate of 74% and a care team–automated alert review rate of 73% within 3 days without incurring the cost of increasing care team staffing was found. Dizziness, nausea and vomiting, and shortness of breath were the symptoms with the highest positive predictive value for adverse events requiring acute interventions. Meaning The findings of this study suggest that technology-enabled monitoring of patient-reported outcomes may provide a useful model for delivering complex care remotely in patients receiving immune checkpoint inhibitors.

completed the embedded algorithm displayed a summary. Patients could also submit symptoms at any point, if desired. If any of the questions met the severity thresholds previously determined, the patient would be prompted to contact their care team with a direct link to phone them.
The MDACC clinical trials platform Prometheus was leveraged as the central source of care team notifications for this trial. A dashboard was created to display summaries of patient reports of alertable symptoms, which were also emailed to their care teams in real time (eAppendix C). Care teams were instructed to review and resolve these alerts within

Methodology applied to identify the set of electronic questions and their initial alert thresholds
The set of symptoms and associated questions that was created to detect upcoming adverse events is included in eAppendix C. A team of clinical experts identified common and/or severe toxicities among this patient group, as well as their corresponding clinical symptoms. Having selected the appropriate symptoms, questions for each were created to capture necessary details to understand when immediate interaction of patients with their care teams was necessary. The PRO-CTCAE 1 provided inspiration but has not been validated for immunotoxicity monitoring nor high-frequency electronic reporting. [2][3][4] To achieve high patient compliance, a team inclusive of clinical content experts, statisticians, user experience experts, and designers arrived at a final set of queries to assure efficiency without compromising goals. First, all symptom survey questions were reviewed, language simplified and shortened whenever possible, and parallel structures maintained across questions to aid comprehension. 5 Second, an initial multiple-choice question was created in which patients selected which symptoms they were experiencing.
This allowed patients to only answer a small subset of questions, i.e., those relevant to their current state.
Uniform and precisely defined symptom queries were central to the study. Symptomatic patients selected from a tailored inventory to inform the care team of toxicity details in a standardized format. The nature and severity of the reported toxicity was used to prompt alerts and recommend consensus remedies (eAppendix D). The initial thresholds for these alerts (eAppendix C) were set based on clinician consensus (e.g., reporting of severe or very severe cough) or a state known to lead to a severe symptom within two weeks (e.g., reporting of fever). This heightened level of granularity and frequency of symptom collection is not often available. Thus, dynamic adaptation of these alert thresholds was planned as described below.
To allow for unanticipated patient concerns, provisions were made to encourage patients to write in any additional symptoms in free text form. All text entries triggered an alert for care teams to review, as clinical severity could not be determined in real time.

Data collection
Patient answers and care team responses were collected in real time and monitored weekly by trial staff. Research teams also graded adverse events using CTCAE guidelines 6 via chart review every four to six weeks. A monthly cadence was used to extract dates and details of clinical interventions (defined above). Assignment of interventions to adverse events was manually adjudicated by the principal investigator while blinded to concurrent symptom responses.

Consensus events meriting alerts
The goal of the monitoring system was to alert the care team to acute, clinically meaningful toxicities. We defined these adverse events as those with Common Terminology Criteria for Adverse Events (CTCAE) grade of at least 2 and that were associated with a clinical intervention. Clinical interventions were defined as treatment modifications (pause, dose reduction, or termination of immunotherapy, concomitant targeted therapy, or supportive therapy) or new interactions with healthcare providers due to toxicities that developed after treatment initiation, including emergency department visits or hospitalizations. Adverse events within two weeks of an alert were considered temporally linked to that alert.
The symptoms included in this study were those judged by the panel of experts, described below, to be of significant concern to warrant alerting care teams and patients for potential intervention. The electronically enabled alerts were considered appropriate when they were followed by a linked intervention as defined above. Although beyond the scope of this initial report, effective alerts were those that led to remedies that mitigated toxicity.
Robust analysis of effectiveness, using this definition, requires longer follow-up and will be the subject of a subsequent report.

Dynamic alert thresholds
Operating characteristics were estimated for each symptom alert and were conditional on the symptom having been reported. To prevent over-indexing for patients with more alerts, subsequent reports for a patient were downweighted by a factor of 1/k, where k is the number of alerts [for positive predictive value (PPV) and sensitivity] or non-alerts [for negative predictive value (NPV) and specificity] for that symptom for that patient. The formulas used for PPV, NPV, sensitivity, and specificity are listed in eAppendix F.
The analysis included stringent requirements for threshold increases to ensure patient safety, reduce risk of overfitting, and account for the fact that a single symptom alert could be triggered by one of several associated questions. Symptoms with a sensitivity smaller than 10% were identified as candidates for an increase in threshold stringency. Within each symptom, questions that had led to alerts in at least five unique patients were considered, with those more commonly alerting studied first. The impact of increasing this initial question's alert threshold by one level was studied by calculating the NPV and sensitivity of the full symptom alert under this counterfactual condition. The change was recommended if the NPV increased and the sensitivity did not decrease (i.e., none of the alerts removed were linked with subsequent adverse events). If this change was found to be beneficial, the subsequent level was considered, as well as the next most common alerting question for that symptom. This procedure continued until a question did not lead to a change, at which point no further questions were considered for that symptom and the analysis proceeded to the next symptom. Recommended changes were then reviewed and approved by clinical experts before implementation.

Diagnostic evaluation:
For all potential immune-related adverse events send the following blood tests: • Complete blood count (CBC) with differential, comprehensive metabolic panel The following section includes definitions of toxicity grades using Common Terminology Criteria for Adverse Events (CTCAE) version 5.0. In addition, this section contains "triggers" for initiation of management for these toxicities based on "standard" recommendations and our clinical experience.