Efficiency of Electronic Health Record Assessment of Patient-Reported Outcomes After Cancer Immunotherapy

Key Points Question Can an electronic patient-reported outcome (ePRO) model improve outcome in cancer immunotherapy compared with a traditional model? Findings In this randomized clinical trial of 278 patients from 28 Chinese hospitals comparing ePRO and traditional follow-up models, the intervention group showed a reduced incidence of serious immune-related adverse events, fewer emergency department visits, a lower rate of treatment discontinuation, a higher quality of life, and less time needed to implement the model. Meaning These findings suggest that the ePRO follow-up model is efficient and may provide reliable information and management recommendations in patients receiving immunotherapy for cancer.


Introduction Cancer Immunotherapy and Immune-Related Adverse Events
The rapid advances in cancer immunotherapy using immune checkpoint inhibitors (ICIs) have led to significantly improved survival of patients. 1,2At the same time, ICIs have been associated with multiple immune-related adverse events (irAEs). 3The irAEs can affect a wide range of organs and induce nonspecific symptoms with delayed onset 4 and prolonged duration that are easily neglected, which may lead to life-threatening disorders.The incidence of irAEs of any grade attributable to single-agent ICI therapy is as high as 90%. 5,6Among these, irAEs of grade 3 or higher occurred in approximately 20% to 43% of patients, 7 and 2% of the patients died due to irAEs worldwide. 8nce, patients and clinicians should be well informed and vigilant to ensure that irAEs are recognized and interventions are performed in a timely manner.

Follow-up of Patients Experiencing irAEs
Immune-related adverse events are more likely to be experienced outside health care settings and therefore need to be followed up effectively.Because of the large amount of data generated and the long period of time needed, an efficacious, less labor-intensive, time-saving follow-up model is needed.However, the number of studies, especially large-scale randomized clinical trials, are limited.
Hoffner and Rubin 9 reported improvements in telephone triage by using immuno-oncologyessential materials 10 to reduce the burden of irAE follow-up.Le et al 11 evaluated the impact of a pharmacist-led irAE follow-up process that reduced the times required for physicians to manage irAEs and increased physician confidence.
As a new follow-up model, electronic patient-reported outcomes (ePROs), including healthrelated questionnaires completed by patients, could effectively capture symptoms and their severity.
3][14][15] Furthermore, Basch et al 16 found that the ePRO model used for management of patients with cancer could result in improved quality of life (QOL), fewer emergency department (ED) visits, longer duration of palliative chemotherapy, and superior quality-adjusted survival.However, only 1 study 17 assessed an eHealth intervention system used for 50 patients with malignant melanomas receiving cancer immunotherapy.The study reported that the level of satisfaction with eHealth intervention was high among patients and clinicians.
However, the follow-up efficiency of an ePRO model for improving safety for patients receiving cancer immunotherapy is currently under debate, especially for a lack of evidence derived from large sample populations or randomized clinical trials.This study aimed to investigate the efficiency of the ePRO follow-up system for improving the safety and QOL of patients receiving cancer immunotherapy and making the follow-up process less time consuming.

JAMA Network Open | Oncology
Efficiency of a Patient-Reported Outcome Follow-up Model in Cancer Immunotherapy

Participants
Enrollment ended in March 2021 when the last patients had their 6-month follow-up.Eligible individuals for the study were 18 years or older and were receiving cancer immunotherapy; had an Eastern Cooperative Oncology Group performance status of 0 or 1 and life expectancy of at least 6 months; were willing to complete the follow-up process according to the protocol; and could use smartphones or computers to include information in the model application (app) with or without the help of their caregivers.Exclusion criteria consisted of an uncontrolled medical disorder (hepatic failure, kidney failure, respiratory failure, or heart failure), diagnosed mental disease, and refusal to participate.

Interventions Follow-up Team
The ePRO app was built and maintained by 2 computer specialists from the AI Startfish Technology Co, Ltd (D.Z. and S.M.) (eMethods 1 in Supplement 2).A follow-up team that included an oncology specialist and 2 nurses was established in every center.Every center had a unique account for the ePRO app, and each team was responsible for follow-up to their own patients.All researchers had at least 2 years of experience in ICI therapy and received training about the research program, use of the app, and the follow-up process before the study commenced.

Control Group
Researchers educated patients and their caregivers about immunotherapy and common symptoms of irAEs at baseline.Patients were followed up using a traditional model, including clinic visits every 21 days and via the telephone every 3 months.Patients could visit the clinic when they felt any discomfort.

Intervention Group
Patients in the intervention group were registered and followed up via the ePRO app.[19][20] Patients checked the questionnaire weekly and uploaded pictures of examination results between visits on their mobile device or computer.The app sent 3 reminder messages on 3 consecutive days to patients who did not reply.If no reply was obtained, follow-up by telephone was conducted.
9][20] (eMethods 2 in Supplement 2).When grade 1 or 2 irAEs occurred, the standardized advice from the irAE management guidelines 3,[17][18][19][20] was sent to patients via the app automatically.If grade 3 or 4 irAEs were reported, the app alerted the health care team via text message, email, and app simultaneously.The health care team followed up via telephone and comprehensively evaluated the

JAMA Network Open | Oncology
Efficiency of a Patient-Reported Outcome Follow-up Model in Cancer Immunotherapy patient's situation.If serious irAEs were still suspected, patients were advised to go to the hospital or ED for evaluation and treatment.In addition, patients could consult the team on the app anytime (Figure 1).

Relative Indexes of irAEs
We collected data regarding irAE incidence and grades, the rate of occurrence of grade 3 or 4 irAEs (number of cases, with the irAEs divided by the total number of cases), ED visits, and rate of treatment discontinuation and death owing to irAEs.These data were compared between the intervention and control groups.

Quality of Life
Quality of life was assessed using the European Organization for Research and Treatment of Cancer QLQ-C30 validated by Aaronson et al 21 and used to measure the QOL of patients every 3 months.The QLQ-C30 was translated into Chinese by Wan et al 22 and consisted of 30 items, including 5 functional dimensions (physical, role, emotional, perception, and social functions), 9 symptom dimensions, and 2 health situation dimensions.The 2 health situation items were assigned scores ranging from 1 to 7, whereas the other items were evaluated using a 4-grade Likert score.Based on the rules for assigning scores, we transformed every dimensional score to a standard score ranging from 0 to 100.The functional dimension score and the health situation score were positively correlated with patient QOL.The control group received the traditional model of follow-up; the intervention group received the electronic patient-reported outcome (ePRO) model, follow-up monitoring, and consultation via the ePRO application (app).irAE indicates immune-related adverse effect; serious irAEs include grades 3 to 4.

Sample Size
We used PASS software, version 10.0 (NCSS Statistical Software), to calculate the sample size.It was assumed that the incidence of severe adverse reactions in the intervention group would be 10% lower than that in the control group.The unilateral z test was applied; α = .05and β = 0.1.The results showed that 103 patients were needed in each group; considering loss to follow-up, the total sample included 300 cases.

Randomization
Randomization occurred immediately after a participant provided written informed consent at each center.Participants were randomly assigned to the control or intervention group using Interactive Response Technology software, version 3.0 (Almac Clinical Technologies).Each patient's group randomization was based on a site-specific, sequentially numbered study identification number at enrollment.Randomization was stratified based on study site to address potential center effect or bias.This protocol helped to mitigate biases that might stem from differences in patient populations, care management, and other contextual factors that may be unique to an individual study site.

Blinding
Patients were not blinded to their group assignment.To mitigate potential bias caused by nonblinding, we chose objective indicators such as irAEs and the QOL questionnaire as the primary outcome measures for their excellent interrater reliability, external validity, and minimal vulnerability to rater bias. 21,22

Statistical Analysis
All data were input into SPSS Statistics software, version 20.0 (IBM Corp), by 2 researchers (L.Z. and D.Z.).Descriptive statistics were used for the demographic characteristics, clinical conditions, irAE relative indexes, and QOL.Enumeration data were expressed as frequency and percentage, whereas quantitative data were expressed as mean (SD).Variance independent-sample t test and χ 2 test were used to compare the differences in enumeration and quantitative data, respectively, between groups.Two-sided P < .05indicated statistical significance.

Outcomes and Estimation
Relative Indexes of irAEs

Time Expended
During the entire follow-up process, the mean time expended for each follow-up for the intervention and control groups ranged from 0 to 120 and 10 to 100 minutes, respectively.The mean (SD) times were 8.

Discussion
The optimal treatment of patients receiving ICIs could be achieved via the comprehensive assessment, grading, and long-term surveillance of symptoms.An efficient follow-up process that facilitates the monitoring and reporting of irAEs in a timely manner is important for ensuring patient safety.Previous studies have shown that ePRO follow-up enables timely and continuous collection of information regarding symptoms in a cost-effective manner [9][10][11][12][13][14] and results in an improved QOL, fewer ED visits, longer duration of palliative chemotherapy, and superior quality-adjusted survival. 15,16However, the efficacy of ePRO follow-up has not been examined in patients treated with ICIs.The purpose of this study was to investigate the efficiency of the ePRO model in a randomized clinical trial in a mixed sample from multiple centers.

Safety
Our findings show that patients in the intervention group had a trend of fewer severe irAEs, fewer ED visits, less treatment discontinuation, and lower death rates, although no statistically significant differences were observed.Because of the nonspecific performance and delayed onset, detection of irAEs was crucial for ensuring patient safety.Compared with routine follow-up, weekly ePRO follow-up was performed more frequently.It can facilitate an early response to patient symptoms, thereby preventing occurrence of severe irAEs.In addition, severe irAEs are also nonspecific irrespective of their level of lethality, and the ePRO follow-up system can recognize signs of irAEs and

Patient QOL
At 6 months after the intervention, the overall QOL, physical functions, and emotional functions of patients in the intervention group were higher than those in the control group.These findings were similar to those of a previous study. 16One of the possible reasons included the periodic and timely follow-up process that may help to reduce the occurrence of irAEs and increase the advice of safe care for mild irAEs that may alleviate patient symptoms, thus improving physical function.In addition, medical team responses (via standardized advice, telephone, or online consultation) to patients' symptoms in a timely manner may have helped to relieve patient anxiety, depression, or

Time Expended
To further explore the efficiency of the ePRO model, we also compared the time necessary to use it and the routine follow-up model, which has not been compared in previous studies.The results show that the ePRO model was less time consuming than the traditional follow-up process.One possible reason for this difference could be that the ePRO follow-up model, which is based on the information system, mainly uses the platform to automatically send follow-up tasks and standardized advice.
Only a small group of patients who did not respond or were suspected to have serious irAEs were followed up via telephone.Compared with the traditional telephonic follow-up process for each patient, the ePRO approach undoubtedly saves time and labor required for recording, planning, and reviewing the follow-up arrangements for patients, especially with mobile phone connectivity issues, language barriers, hearing loss issues, and communication-related difficulties.
In addition, the system can simultaneously perform automatic recognition of symptoms, assessment of examination results, and timely warnings, thereby reducing the time required by medical staff to check the test results.Given the large clinical workload and shortage of nursing staff, the ePRO follow-up model can effectively reduce follow-up time and improve work efficiency.

Limitations
Our study has some limitations.First, the follow-up period was only 6 months or until discontinuation of treatment, so the long-term outcome of overall and progression-free survival could not be observed.Second, it is vital that clinicians log into the system and see the patients' reports before every consultation.The ePRO app must be implemented in such a way that the process is embedded as part of routine care.In this regard, it is important that the ePRO app is easily accessible to medical staff (ie, integrated in the electronic health record) to be successful.Third, some patients' adherence decreased over time, so the methods of improving adherence need to be considered in future research.

Conclusions
To the best of our knowledge, this is the most up-to-date multicenter randomized clinical trial with a large sample to assess the efficiency of a follow-up model for patients receiving ICIs.Our findings show that the ePRO follow-up model helped to improve the safety and QOL of patients receiving immunotherapy and reduced follow-up time.This novel approach may provide reliable information and management recommendations.

2023 Mean Time per Follow-up Session
Efficiency of a Patient-Reported Outcome Follow-up Model in Cancer Immunotherapy Researchers recorded the time spent on each follow-up session.The mean time spent on follow-up (total time divided by the number of follow-up sessions per patient) was compared between the intervention and control groups.
JAMA Network Open.2022;5(3):e224427.doi:10.1001/jamanetworkopen.2022.4427(Reprinted) March 31, 2022 4/11 Downloaded From: https://jamanetwork.com/ on 09/30/ A total of 364 patients were screened for eligibility; of these, 35 did not meet the inclusion criteria and 29 declined to participate.The remaining 300 eligible patients agreed to participate.The participant flow diagram is presented in Figure2.Among these, 150 patients were randomized to the control group and 150 to the intervention group.Data from 9 patients from the intervention group and 13 from the control group were excluded from the analysis owing to insufficient data.

on 09/30/2023 alert
medical personnel immediately after a patient provides a report.Patients are then contacted and alerted in a timely manner, which makes it possible for a patient to undergo treatment and avoid fatal consequences.Moreover, patients should discontinue immunotherapy immediately after the occurrence of severe irAEs.Early response to mild irAEs can prevent the aggravation of symptoms and the occurrence of severe irAEs.Hence, a decrease in the rate of treatment discontinuation might help improve disease progress.

Table 2 .
irAE Relative Indexes After Intervention a Percentages have been rounded and may not total 100.

Table 3 .
QOL Between Groups Before and After Intervention