We created a multivariable logistic regression model separately within each surgical service adjusting for age, American Society of Anesthesiologists score (1-2 vs 3-4), body mass index, sex, and operative time. We assessed heterogeneity in the adjusted odds ratio (OR) within each service using a fixed-effects meta-analysis across primary service (Cochran Q; P = .39). RT indicates Recovery Tracker.
eAppendix. Recovery Tracker System Details
eTable 1. Included Surgical Services and Most Common Procedures
eTable 2. Recovery Tracker Survey Questions and Alert Thresholds
Customize your JAMA Network experience by selecting one or more topics from the list below.
Identify all potential conflicts of interest that might be relevant to your comment.
Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.
Err on the side of full disclosure.
If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.
Not all submitted comments are published. Please see our commenting policy for details.
Simon BA, Assel MJ, Tin AL, et al. Association Between Electronic Patient Symptom Reporting With Alerts and Potentially Avoidable Urgent Care Visits After Ambulatory Cancer Surgery. JAMA Surg. 2021;156(8):740–746. doi:10.1001/jamasurg.2021.1798
Is electronic symptom reporting with clinical alerts (the Recovery Tracker) for 10 days after ambulatory cancer surgery associated with a reduction in potentially avoidable urgent care visits, defined as an urgent care visit without an admission?
In this cohort study of 7165 patients, implementation of the Recovery Tracker was associated with significantly lower risk of an avoidable urgent care visit and a modest increase in the number of nursing calls.
Electronic symptom reporting with nursing follow-up for clinical alerts may reduce potentially avoidable urgent care visits, supporting broader implementation.
Increasingly complex surgical procedures are being performed in the outpatient setting, increasing the burden on patients and caregivers to manage their postoperative symptoms. Electronic patient-reported symptom tracking may reduce this burden and help patients distinguish between expected symptoms and those requiring intervention.
To determine whether electronic symptom reporting with clinical alerts for 10 days after ambulatory cancer surgery is associated with a reduction in potentially avoidable urgent care visits, defined as a visit not leading to admission.
Design, Setting, and Participants
This retrospective cohort study was conducted at the Josie Robertson Surgery Center (JRSC), Memorial Sloan Kettering Cancer Center’s ambulatory surgery center with overnight stay capacity from September 20, 2016, to December 31, 2018. Patients undergoing prostatectomy, nephrectomy, mastectomy with or without immediate reconstruction, hysterectomy, or thyroidectomy at the surgery center before (n = 4195) and after (n = 2970) implementation of the Recovery Tracker (RT) electronic postoperative symptom survey were included. Data analyses were conducted from February 1 to November 24, 2020.
A short electronic survey assessing symptoms daily for 10 days after surgery, administered via the patient portal, with alerts to the clinical team and follow-up for concerning responses.
Main Outcomes and Measures
The main outcome was Memorial Sloan Kettering urgent care center visits with and without readmission and any readmission within 30 days after surgery. Nursing workload was measured by patient phone calls, emails, and secure messages as documented in the electronic medical record.
A total of 7165 patients were analyzed, including 4195 (median age, 53 [interquartile range (IQR), 44-63] years; 3490 women [83%]) from the pre-RT implementation period and 2970 (median age, 56 [IQR, 46-65] years; 2221 women [75%]) from after full implementation. On multivariable, intent-to-treat analysis by study period, having surgery in the post-RT period was associated with a 22% decrease in the odds of an urgent care center visit without readmission (OR, 0.78; 95% CI, 0.60-1.00; P = .047). Having responded to at least 1 survey was associated with a 42% reduction in the odds of an urgent care center visit without readmission (OR, 0.58; 95% CI, 0.39-0.87; P = .007). There was no change in the risk of admission. Nursing calls increased by a mean of 0.86 (95% CI, 0.75-0.98) calls per patient after RT implementation (P < .001), a 34% increase.
Conclusions and Relevance
In this cohort study, electronic symptom reporting with nursing follow-up for clinical alerts was associated with a reduction in potentially avoidable urgent care visits. The low risk and high benefit of this intervention suggest that these systems should be more broadly implemented.
Increasingly complex surgical procedures are being performed with decreasing lengths of stay and in the ambulatory setting.1-3 Although a short hospital stay has many benefits, it also places additional burdens on patients and caregivers to manage their recovery at home.4 Postoperative symptoms are common and represent a major source of morbidity and distress for patients,5,6 and it can be difficult for patients to distinguish between symptoms that are expected during recovery and potentially serious events, such as infection.7-9
The gap between patients at home and their clinical teams can be bridged by remote, electronic symptom monitoring using patient-reported outcomes, promoting patient-centered care, communication, and shared decision-making.10-15 More broadly, patient-reported outcomes have been shown to improve symptom control and quality of life and even prolong survival rates in patients with cancer.10,16-20 Pilot programs involving acute symptom reporting after surgery with actionable alerts have reported high patient acceptance and promising results in reducing patient-reported symptom burden.12,21-23 The effect of such programs on the need for acute care is not well characterized.
An acute symptom reporting system was recently developed and implemented at the Josie Robertson Surgery Center within Memorial Sloan Kettering Cancer Center (MSK). The Josie Robertson Surgery Center is an ambulatory facility that performs more complex cancer surgical procedures than most surgical centers, in part because of its overnight stay capacity.24 The Recovery Tracker (RT) system, developed to facilitate monitoring of potentially concerning postoperative symptoms, is a brief electronic survey sent daily to patients via the MyMSK Patient Portal for 10 days to assess common postoperative symptoms. Each response has preset thresholds which, if exceeded, automatically alert the surgeon’s care team to follow up with the patient. Recovery Tracker was implemented incrementally by service as part of routine care beginning in late 2016 and was fully rolled out for all qualifying procedures by December 2017.
We hypothesized that interactions with the care team triggered by symptom alerts would help patients identify which symptoms they could manage at home compared with those that might require hospital admission. We therefore examined unplanned MSK urgent care center (UCC) visits within 30 days after surgery before and after RT implementation to determine whether there was a reduction in visits that did not lead to an admission, what we defined as a potentially avoidable UCC visit.
After approval by the MSK institutional review board, which waived the need for consent because the study was considered low risk, we performed a retrospective cohort analysis of MSK UCC visits and readmissions within 30 days of undergoing a procedure at the Josie Robertson Surgery Center, including all procedures for patients ultimately eligible to receive RT (eTable 1 in the Supplement). Only the first procedure was included for patients with multiple visits; patients who received a modified feedback report after completing RT as part of a clinical trial25 were also excluded. All data were collected as part of routine clinical care and obtained from the institutional database. This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.
The primary analysis compared the likelihood of MSK UCC visits, with and without hospital readmission within 30 days, and readmission within 30 days (including readmissions directly from home or clinic) before and after RT implementation following intent-to-treat principles. The RT survey went live at different times for each surgical service, beginning with urology in September 2016 and head and neck surgery being the final addition in December 2017. Patients were considered to be in the pre-RT study period if the surgery occurred before the go-live date for their service. The post-RT period was defined as any surgery occurring from January 1 through December 31, 2018, after RT was available to all services.
As a sensitivity analysis, we tested the association between response to at least 1 survey and the risk of a UCC visit and/or readmission (“responder analysis”) among patients treated in 2018. Only patients who completed a survey within the first 10 postoperative days and before any UCC visit or readmission were defined as responders in this analysis.
Recovery Tracker is a short, electronic survey assessing symptoms for 10 days after surgery (eAppendix in the Supplement). Briefly, patients receive daily emails with a link to an online questionnaire that includes items on a variety of postoperative symptoms, such as pain, fatigue, and shortness of breath. The RT survey items (eTable 2 in the Supplement) were adapted from the National Cancer Institute Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events,26 a validated instrument designed for standardized symptom assessment in cancer patients for clinical trials. The clinical team was alerted automatically if a symptom reached a prespecified threshold, which varied depending on the symptom and the time since surgery.27 For a “yellow alert,” denoting intermediate-level symptoms such as severe pain, an automatic notification was sent to the surgeon’s office practice team, and the patient was assessed by telephone or portal secure message during regular office hours. For “red alerts,” for concerning symptoms such as very severe pain or severe shortness of breath, an electronic pop-up instructed the patient to contact the doctor immediately, and the surgeon’s office was notified.
Analyses were conducted from February 1 to November 24, 2020. Our primary objective was to assess whether the odds of a potentially avoidable urgent care visit, defined as an MSK UCC visit without readmission within 30 days of surgery, differed between the period before vs after RT implementation. Testing for time-based trends in outcomes in the preimplementation period using multivariable logistic regression adjusting for age, American Society of Anesthesiologists physical status (ASA score; 1-2 vs 3-4), body mass index, sex, surgical service, and operative time did not reveal evidence of a trend for any outcome (all P values ≥.087), so data were not corrected for secular trends. For our primary analysis following intent-to-treat principles, we tested the association between the outcomes of interest and study period using multivariable logistic regression adjusting for the above covariates. We repeated this analysis, testing the association between outcomes and responder status among patients treated in 2018 as a sensitivity analysis. To assess whether the effect of implementation of RT on risk of a potentially avoidable UCC visit differed by surgical service, we tested for heterogeneity in the adjusted odds ratio within each service, adjusting for the aforementioned covariates except sex (as within the gynecologic surgery service there were no male patients), using a fixed-effects meta-analysis. All analyses were repeated for the outcomes of UCC visit with readmission and any readmission, both within 30 days of surgery.
To assess whether implementation of RT and its alert-triggered patient follow-ups influenced clinic nurse workload, we examined nurse-patient communications, including telephone calls, emails, and portal secure messages as documented in the electronic medical record (“nursing calls”). We tested for an association between study period and the number of patient communications within 30 days of surgery using multivariable negative binomial regression adjusting for the aforementioned covariates. P values were 2-sided and were calculated by Kruskal-Wallis or χ2 tests. P < .05 indicated significance. All analyses were conducted using R, version 3.6.1 (R Foundation).
After retaining only a patient’s first RT-qualifying procedure and excluding 569 patients participating in a conflicting clinical study,25 a total of 7165 patients were included: 4195 patients (median age, 53 [interquartile range (IQR), 44-63] years; 3490 women [83%]) were analyzed from the pre-RT implementation period and 2970 (median age, 56 [IQR, 46-65] years; 2221 women [75%]) from calendar year 2018, after RT was fully rolled out among all services. Because RT was implemented sequentially by service, services that began RT earlier stopped contributing patients to the pre-RT group sooner and thus are underrepresented compared with the post-RT group. Thus, the many differences in patient characteristics between these groups (Table 1) reflect differences in case mix, particularly with regard to sex, robotic-assisted procedure, operating room time, and overnight stay (Table 1). The 2970 patients in the post-RT cohort were classified as nonresponders (n = 979) or responders (n = 1991) for the sensitivity analysis according to completion of at least 1 survey as described above. The differences in case mix and characteristics between these groups therefore follow the differences in responder rates among services, with urology having the highest responder rate (492 of 670 [73%]) and plastic surgery the lowest (224 of 421 [53%]).
Among the 2970 patients treated in the post-RT period, 2487 patients (84%) were enrolled in the MyMSK patient portal and thus offered RT surveys. Of these 2970 post-RT patients, 2000 (67%) submitted at least 1 RT survey, and 1434 (48%) submitted at least 4. Among the patients who responded to at least 1 survey, the median number of surveys answered was 6 (IQR, 3-8), and each survey took a median of 1.5 (IQR, 1.1-2.2) minutes to complete.
Among the 1991 patients that satisfied our “responder” definition for the sensitivity analysis, 982 (49%) triggered at least 1 alert. The median number of days with an alert was 1.5 (IQR, 1.0-2.0), of which 69% (1381 of 2004 days) included only yellow alerts and 31% (623 of 2004 days) included red or red and yellow alerts. The most common symptoms generating yellow alerts were pain (369 of all 1992 yellow alerts [19%]), redness (355 [18%]), no bowel movement (247 [12%]), and constipation (205 [10%]) and for red alerts, passing gas (177 of 728 [24%]), fever (112 [15%]), and constipation (88 [12%]).
On multivariable analysis, having surgery in the post-RT period was associated with a 22% reduction in the odds of a UCC visit without readmission (OR, 0.78; 95% CI, 0.60-1.00; P = .047 (Table 2 and Table 3). Only 2 covariates were associated with a change in risk of a UCC visit; operating room time was associated with an increased risk (OR, 1.15; 95% CI, 1.02-1.29; P = .02), and plastic/reconstructive service was associated with a decreased risk (OR, 0.57; 95% CI, 0.36-0.88; P = .01) (Table 2). Despite the wide range of risk of a UCC visit without readmission across services, we did not find evidence to suggest heterogeneity in the RT-associated change in risk among services (Figure). The absolute risk reduction (Table 4) for RT was 0.9% (95% CI, 0.0%-1.8%), meaning that 1 patient avoided a UCC visit without readmission for every 111 offered the RT (number needed to treat). In contrast, undergoing surgery in the post-RT period was not associated with a change in the risk of a UCC visit with readmission or any readmission (Table 3 and Table 4).
Responder status as defined above was associated with a 42% reduction in the risk of a UCC visit without readmission (OR, 0.58; 95% CI, 0.39-0.87; P = .007 (Table 3). This outcome corresponds to a 1.8% (95% CI, 0.4%-3.5%) absolute risk difference and number needed to treat of 56. We did not find evidence of difference in the risk of a UCC visit with readmission or any readmission by responder status (Table 3), although confidence intervals were relatively wide (Table 4).
The predicted number of nursing calls per patient within 30 days was 2.5 for those treated pre-RT and 3.4 for those treated post-RT. This difference corresponds to an adjusted increase of 0.86 (95% CI, 0.75-0.98) calls (P < .001) or 34%.
In this retrospective cohort study, electronic patient symptom reporting with clinical alerts after ambulatory cancer surgery was associated with a significant decrease in urgent care visits that did not result in readmission within 30 days postoperatively. Importantly, we did not find an association with readmissions, suggesting that the reduction in avoidable visits was related to the intervention and not secular trends, such as improved perioperative technique. Moreover, this association provides evidence that the RT does not create a false sense of security and delay treatment of important problems. Our findings were robust across 5 surgical services performing a wide range of procedures and were of clear clinical relevance: for 111 patients offered the RT, 1 UCC visit without admission can be avoided. The change in clinic nurse workload associated with this benefit was an increase of less than 1 additional call per patient.
From the patient and caregiver perspectives, returning to the hospital or an emergency department after surgery can be extremely stressful, both in making the decision to seek acute care and in the challenges inherent to making such a visit, particularly if it turns out to be unnecessary. In fact, reducing potentially preventable acute care visits for chemotherapy-related symptoms is the first cancer quality metric in the Hospital Outpatient Quality Reporting Program,28 a measure intended to reduce the impact on quality of life, health care utilization, and costs of predictable treatment side effects that could be avoided if identified early and managed proactively.29 For our purposes, we defined a potentially preventable UCC visit as a UCC visit for any reason that did not result in hospital admission. Not all UCC visits without admission are avoidable—for example, replacing a clogged urinary catheter—but this simple classification provides a reasonable surrogate.
The number needed to treat to reduce unnecessary UCC visits among responders was lower than that found in the intention-to-treat analysis. Differences between patients or caregivers who choose to respond to an electronic survey and those who do not likely include confounding factors that could be associated with better outcomes, including higher health literacy, socioeconomic status, and education. These factors could lead to consequent higher health care engagement and comfort managing a problem at home, avoiding an unnecessary UCC visit. Although such biases might account for some of the increased effect size, 51% of nonresponders were signed up for the portal, so access to technology was not limiting in this group. Hence, efforts to improve response rates seem likely to increase the positive outcomes of the RT.
This study adds to the small body of research on the influence of patient-reported symptom monitoring in surgical patients. In a prospective, randomized trial, Cleeland et al12 studied twice-weekly automated telephone symptom reporting for 4 weeks after thoracic surgery, with or without clinical alerts with nursing follow-up for severe symptoms, similar to RT. The alert group in this small study reported greater reductions in symptom severity and interference, suggesting that alert-triggered interactions with the nursing team, and not the survey itself, was beneficial. These authors did not examine the need for acute care. Borsuk and coauthors30 assessed the effect of an advanced text-based monitoring system with escalating alerts on the need for acute care in 281 patients after colorectal surgery, using a preimplementation and postimplementation design similar to ours. They found a reduction in both ER visits and readmissions in the postimplementation group but did not see a difference in avoidable ER visits as defined by chart review. This finding may have been related to the low sample size, which was about 30 times smaller than the current study’s and included only 2 events.
Approximately one-half of patients responding to surveys triggered alerts, the majority of which were for moderate to severe symptoms expected after surgery, such as pain, constipation, and wound condition. These alerts triggered communication from the office practice nurses and generally resulted in reassurance, reinforcement of postoperative instructions, and education, with rare escalation to the surgeon or referral to the UCC for evaluation. Because these proactive communications are the main difference in the course of pre- vs post-RT patients, it appears likely that this context-sensitive support is at least in part responsible for the reduction in avoidable UCC visits. In addition to the reassurance, management tips, and triage provided by the nurses, the survey itself might contribute to increased comfort managing symptoms at home by increasing patient awareness of their symptoms, reinforcing that these symptoms are expected and increasing the sense of connection to the clinical team. Future work should explore the relationship between specific symptoms and outcomes as well as factors contributing to the observed reduction in UCC visits.
Certainly, the retrospective nature of this study requires cautious interpretation with regard to causality. Although this observational study cannot definitively indicate causality, a causal role of RT in reducing potentially avoidable UCC visits is supported by the observed association between RT and our hypothesized outcome but not other outcomes (eg, readmission), the stronger association in the analysis of responders compared with intent-to-treat, and the lack of evidence of secular trends. Furthermore, although there were differences in patients who did and did not receive RT, these differences were all biased against RT. In Table 2, the only covariates that were associated with the end point were operating room time (increased risk) and plastics service (decreased risk). The group that received RT had longer operating times and included fewer patients who received plastic/reconstructive surgical care.
Implementation of a postoperative symptom monitoring system has both benefits and drawbacks. In qualitative interviews of patients conducted as part of a related clinical trial,25 participants uniformly liked the system, found it easy to use, and reported that it made them feel more confident, aware of what to look for, and connected to their clinical team. While initial implementation may be expensive, once the system is operational, the marginal cost of distribution to additional patients is very low. That said, the workload of additional nursing calls is nontrivial; efforts to refine alert thresholds, for example, may reduce this incremental workload while maintaining the observed benefits. However, given the stress and inconvenience of an unnecessary UCC visit for patients and caregivers, we believe that, even in its present state of development, the RT adds value both for the patient and care team. Future enhancements should focus on improving this value proposition, improving the patient experience and outcomes, and reducing the workload for the clinical team.
In addition to the retrospective study design, there are important limitations to our study. We could capture only MSK UCC visits and not acute care visits to other sites, so these data represent an underestimate of total acute care visits. However, we have no reason to suspect that behaviors in seeking acute care would differ among the study groups. Although undergoing relatively complex cancer surgeries for the ambulatory setting, the patients in this study were generally healthier and had fewer complications than those undergoing inpatient surgeries. Whether the benefits of RT translate to the inpatient setting remains to be seen, although the potential benefits would also be greater for patients undergoing more major procedures. In addition, as mentioned above, our high portal participation rates may reflect these patients’ higher-than-average socioeconomic status, education, and engagement in their care, potentially limiting applicability to other settings.
Following the introduction of electronic patient-reported symptom monitoring with clinical alerts and follow-up for concerning responses into routine care for patients undergoing ambulatory cancer surgery, we observed a significant decrease in potentially avoidable urgent care visits (ie, not resulting in readmission). Future work is required to explore the predictive value of specific alerts and the influence of similar interventions on patients undergoing more complex surgical procedures; however, the low risk and high benefit of this system suggest that symptom monitoring with follow-up should be more broadly implemented.
Accepted for Publication: February 26, 2021.
Published Online: June 2, 2021. doi:10.1001/jamasurg.2021.1798
Corresponding Author: Brett A. Simon, MD, PhD, Josie Robertson Surgery Center, 1133 York Ave, New York, NY 10065 (email@example.com).
Author Contributions: Dr Simon had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Simon, Assel, Baron, Cracchiolo, Vickers, Laudone.
Acquisition, analysis, or interpretation of data: Simon, Assel, Tin, Desai, Stabile, Cracchiolo, Twersky.
Drafting of the manuscript: Simon, Assel, Baron, Cracchiolo, Vickers.
Critical revision of the manuscript for important intellectual content: Simon, Assel, Tin, Desai, Stabile, Twersky, Vickers, Laudone.
Statistical analysis: Simon, Assel, Tin, Vickers.
Administrative, technical, or material support: Simon, Desai, Stabile, Baron, Cracchiolo, Twersky.
Supervision: Simon, Twersky, Laudone.
Conflict of Interest Disclosures: None reported.
Funding/Support: This study was supported in part by a National Institutes of Health/National Cancer Institute Cancer Center Support Grant to MSKCC (P30 CA008748) and a Patient-Centered Outcomes Research Institute (PCORI) research award (IHS-1601-34355) (Dr Simon, Dr Vickers, Ms Assel, and Ms Stabile).
Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Additional Contributions: The authors thank Jessica Moore for editorial assistance and Taylor McCready, BA, and Olga Strachna, MS, for data analytics. Andrea Pusic, MD, was an invaluable contributor to the conception and implementation of the Recovery Tracker. Special thanks to Beau Amaya, BSN, RN, OCN, Christine Cingari, MSN, RN, OCN, Anne Collins, RN, Nicole Connors, RN, OCN, Patricia Coyle, MSN, RN, Connie Estes, RN, Jericho Garcia, MSN, RN, Sheila Mohan, RN, Maryellen O’Sullivan, MA, RN, Rori Salvaggio, MSN, RN, Chasity Walters, RN, Peter Stetson, MD, Monica Allison, BA, Michael Eubank, BBA, Alyse Kassa, BA, and the many more nursing and informatics staff who contributed to the development, refinement, implementation, and support of the Recovery Tracker platform. None of these individuals received compensation for their contributions.