Development and Validation of a Score to Predict Acute Care Use After Initiation of Systemic Therapy for Cancer

Key Points Question Is it possible to predict which patients with cancer will require early acute care after starting systemic therapy? Findings In a cohort study, using population-based administrative databases in Ontario, Canada, a score was developed and validated to predict a visit to the emergency department or a hospitalization within the first 30 days after starting systemic therapy for cancer based on cancer type, systemic therapy regimen, previous emergency department visits, and age. Meaning The findings suggest that patients at high risk for use of acute care after starting systemic therapy for cancer can be identified using a simple score that is available at the point of care.


Introduction
Use of acute care is common during systemic therapy for cancer.A systematic review estimated that almost half of patients in observational studies are hospitalized while receiving chemotherapy. 1Use of acute care is undesirable because it may reflect suboptimal control of disease-related symptoms or toxic effects of treatment and constitutes the largest component of cancer care costs. 2 The risk of acute care use with cancer treatment should be included as part of the informed consent process because this information can influence treatment decisions. 3Some acute care use is preventable 4 and interventions can reduce acute care use, morbidity, and costs. 5,6Reducing acute care use is a strategic priority of health care policy makers including Cancer Care Ontario 7 (CCO) and the Center for Medicare and Medicaid Innovation. 8In Ontario specifically, CCO plans to develop a "standardized and comprehensive approach to managing treatment-related toxicity," 7 including "a standardized process/tool for stratifying patients by toxicity risk." 9 Strategies to identify patients at risk for acute care use can facilitate research on prevention.
Although several existing risk scores predict specific toxic effects from chemotherapy such as febrile neutropenia, [10][11][12] prediction scores for acute care use have focused only on hospitalizations after starting palliative chemotherapy for solid tumors and have not used population-based data. 13,14[17][18][19] In this study, we developed and validated the Prediction of Acute Care Use During Cancer Treatment (PROACCT) score to estimate the probability of acute care use within 30 days after the initiation of systemic therapy for solid and hematologic cancers using population-based administrative data.Thirty days was selected because most acute care use after initiating systemic therapy for cancer occurs during this period 20 and because early use of acute care may be the most relevant to the design and evaluation of preventive interventions.

Cohort Creation
The cohort consisted of patients in Ontario who initiated an anticancer systemic therapy during the accrual period between July 1, 2014, and June 30, 2015.The cohort was created by linking multiple population-based administrative and clinical databases.These databases were linked using unique, encoded identifiers and analyzed at the Institute for Clinical Evaluative Sciences.Health care professionals and hospitals in Ontario are reimbursed for services contingent on the provision of administrative data.As a result, administrative databases in Ontario capture more than 95% of health care services rendered. 21This study was conducted in accordance with the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) statement. 22The Sunnybrook Research Institute Research Ethics Board approved this study.Analyses were undertaken at the Institute for Clinical Evaluative Sciences, a prescribed entity under Ontario's Personal Health Information Privacy Act.As such, individual consent is not required for the use of data for approved research projects.
Patients were identified using the Activity Level Reporting database maintained by CCO. 23tients were excluded if they were treated for a rare cancer (defined as <20 patients in the development cohort), a myeloproliferative neoplasm, or cancers commonly treated on an inpatient basis, such as acute leukemias.Patients were also excluded if their systemic therapy was started on an inpatient basis, they were concurrently being treated for multiple cancers, they were younger than 18 years, they did not have a valid Ontario Health Insurance number, or they had a missing value for age, sex, or region of residence.Systemic therapy treatment courses were included if they were initiated during the accrual period.Initiation of a new treatment regimen was defined as the first dose of a systemic therapy

Outcomes
The primary outcome was the occurrence of at least 1 emergency department (ED) visit or hospitalization during the first 30 days after the initiation of systemic therapy (AC30), regardless of the reason or duration.Emergency department visits were ascertained from the National Ambulatory Care Reporting System database 27 and hospitalizations from the Canadian Institute for Health Information Discharge Abstract Database. 28

Baseline Characteristics
We generated a list of characteristics that had prior evidence for an association with acute care use based on a systematic review of the literature 29  Hopkins ACG System, Version 10), 32 which incorporate information from multiple databases.
Symptoms were self-reported using the Edmonton Symptom Assessment System (ESAS); scores were dichotomized, where a score of 4 or greater reflected moderate or severe symptoms. 33,34rformance status was self-reported using the Eastern Cooperative Oncology Group (ECOG) scale.
The ESAS and ECOG scores came from the Symptom Management and Reporting Database. 35,36The ESAS and ECOG values included were those closest to the regimen start date, as long as they were recorded within the preceding 30 days.Emergency department visits and hospitalizations within the previous year were identified using the same approach as with AC30.The risk score was based on a multivariable logistic regression estimated in the development cohort with AC30 as the outcome using a backward selection procedure (2-tailed P < .05 was considered significant).Aggregated Diagnosis Groups, income quintile, rural residence, facility level, the receipt of palliative radiotherapy, ESAS score, and ECOG score were excluded from the selection procedure because these characteristics may not be readily available to clinicians and had a limited association with score performance in a sensitivity analysis.Treatment courses were excluded if there was a missing value for a characteristic included in the regression (complete case analysis).

Statistical Analysis
Coefficients from the multivariable regression model were converted into integer "points" to create the score. 37The points for each characteristic are summed to generate an individualized score.
The risk score was assessed in the validation cohort.The predicted probability of AC30 came from a logistic regression using the score from the development cohort.Discrimination was evaluated visually with the receiver operating characteristic curve and statistically using the C statistic. 38Calibration was evaluated visually using a locally estimated scatterplot smoothing curve comparing predicted with observed AC30 proportions. 38Statistical testing for calibration was not performed because the large sample size could make small departures in the observed from predicted values seem significant. 39Decision curve analysis assessed the potential clinical benefits of using the PROACCT score to select patients for an intervention to prevent AC30.Decision curve analysis is described in detail elsewhere. 40In brief, decision curve analysis evaluates the value of a predictive model when making a clinical decision, for instance, selecting patients for a preventive intervention.Three strategies are compared: selecting all patients for the intervention (treat all), selecting no patients (treat none), and selecting patients using the predictive model.In decision curves, the x-axis depicts the threshold probability, which is chosen by the decision-maker.In the context of this study, a decision-maker would compare the strategies at a threshold probability of 0.2 if he or she was willing to apply the intervention to 5 patients if one of them would otherwise have had an AC30 with certainty. 41The threshold probability selected by a decision-maker will depend on the relative costs and benefits of the intervention, which can include health, financial, and other considerations, so a range is provided.The y-axis depicts the net benefit of each strategy, which is expressed in terms of the value of true positives.For example, a net benefit of 0.1 is the value of applying the preventive intervention to 10% of patients, all of whom would have had an AC30 with certainty otherwise.Data analysis was conducted from December 1, 2016, to August 10, 2019.
Analyses were performed in SAS, version 7.1 (SAS Institute Inc).

Sensitivity Analysis
We evaluated whether the addition of facility level, Aggregated Diagnosis Groups, income quintile, ESAS score, ECOG score, and receipt of radiotherapy improved the model.We assessed how the score performed at predicting each individual component of AC30 separately: ED visits that did not lead to hospitalizations, and hospitalizations.We also assessed the score in the subgroups of patients treated with palliative and nonpalliative intent.

Cohort Description
We identified 12 162 patients with treatment initiations in the development cohort and 15 845 patients with treatment initiations in the validation cohort (Table 1).

Development of the PROACCT Score
In univariable regressions in the development cohort, AC30 was significantly associated with all characteristics except sex, rural residence, and curative-intent radiotherapy (Table 2).The C statistic of the full multivariable model in the development cohort was 0.70 (95% CI, 0.69-0.71;P < .001).In the simple multivariable model using backward selection excluding characteristics that might not be readily accessible at the bedside (income quintile, rural residence, radiotherapy, and facility level) or those with frequently missing values (ECOG score and ESAS score), the C statistic was 0.68 (95% CI, 0.66-0.69;P < .001).
The PROACCT score was created using coefficients from the 3 characteristics in the final multivariable model in the development cohort (Table 2).The score ranged from 0 to 13 (Table 3).
The characteristics within the score were available for 11 517 patients (94.7%) in the development cohort and 14 649 patients (92.5%) in the validation cohort.The score captured most of the discrimination of the model, with a C statistic of 0.67 (95% CI, 0.66-0.69;P < .001) in the development cohort (Figure , A).

JAMA Network Open | Oncology
Predicting Acute Care Use After Initiation of Systemic Therapy for Cancer

Validation of the PROACCT Score
In the validation cohort, the PROACCT score had a C statistic of 0.61 (95% CI, 0.60-0.62;P < .001)(Figure , A).The risk of AC30 was significantly associated with the PROACCT score in the validation cohort (odds ratio per point increase, 1.22; 95% CI, 1.20-1.24;P < .001).The score was reasonably calibrated, slightly underestimating AC30 up to a score of 10, and then overestimating for scores of 12 and 13 (Figure, B and Table 4).
Decision curve analysis for the PROACCT score in the validation cohort is displayed in Figure , C. This decision curve demonstrates that selecting patients for an intervention using the PROACCT score had an appreciable net benefit compared with the treat all and treat none strategies for threshold probabilities of approximately 0.2 to 0.35.Therefore, selecting patients for a preventive intervention for AC30 using the PROACCT score provides a net benefit if a decision-maker is willing to apply the intervention to between 3 and 5 patients to include 1 patient who would have had an AC30.

Sensitivity Analysis
We generated a multivariable logistic regression model with backward selection from all available characteristics, including those that might not be readily available to clinicians: Aggregated Diagnosis Groups, income quintile, rural residence, facility level, ESAS score, ECOG score, and the receipt of radiotherapy.The resulting model included the same characteristics as the PROACCT score plus rural residence, Aggregated Diagnosis Groups, and ESAS scores for drowsiness, lack of appetite, shortness of breath, anxiety, and well-being.Discrimination was only slightly improved (development cohort C statistic, 0.69).Given that this model had comparable performance but more complexity and included characteristics that may be difficult to obtain at the bedside, we used the simpler model to generate the score.We also assessed the performance of the PROACCT score for the separate components of AC30.For ED visits that did not result in hospitalizations, the C statistic was 0.60 (95% CI, 0.59-0.61;P < .001) in the validation cohort.For ED visits resulting in hospitalizations or direct admissions, the C statistic was 0.59 (95% CI, 0.56-0.62;P < .001) in the validation cohort.We also assessed the score in palliative treatment-intent and nonpalliative treatment-intent subgroups, where the C statistics in the validation cohorts were 0.64 (95% CI, 0.63-0.65;P < .001)for the palliative treatment-intent cohort and 0.60 (95% CI, 0.58-0.61;P < .001)for the nonpalliative treatment-intent cohort.

Discussion
In this study, we developed and validated the PROACCT score to predict AC30.The score was developed using administrative and clinical data from the population of Ontario that captured a rich set of predictive characteristics including the specific treatment regimen and patient-reported  These results suggest that characteristics observed in our data may explain only a portion of acute care use, with the remainder owing to randomness; unavailable characteristics such as laboratory test data 14 ; difficult-to-measure characteristics such as patient beliefs, 42,43 self-management skills, and behaviors 44 ; or the availability of other health system supports. 5,6e score was reasonably calibrated in a geographically distinct validation cohort.Calibration is often underappreciated 45 but is important for decision-making by patients, clinicians, and policy makers.For instance, a recent survey of patients with cancer found that information on the risk of acute care contributes to patient decision-making regarding chemotherapy. 3Moreover, the PROACCT score provided a net benefit across reasonable probability thresholds using decision curve analysis, which incorporates both discrimination and calibration. 41Decision curve analysis demonstrated that using the score to select patients for a proactive preventive intervention would be beneficial if a decision-maker was willing to apply the intervention to between 3 and 5 patients to include 1 patient who would have AC30.This range may include the cost-benefit ratios for many interventions such as proactive symptom monitoring using electronic or interactive smartphone applications. 46,47evious studies generated models to predict hospitalizations and toxic effects of chemotherapy based on smaller samples.1][12] For instance, Lyman et al 12 developed a logistic regression model to predict neutropenic complications in 3760 patients from 115 centers in the United States, with a C statistic of 0.805 using a 2:1 random split for validation.

JAMA Network Open | Oncology
To our knowledge, the PROACCT score is the first to predict both ED visits and hospitalizations in a diverse cohort of patients across all stages of disease including solid and hematologic malignant neoplasms, derived from population-based administrative data.

Limitations
Our study should be interpreted in the context of its potential limitations.First, while we validated the score using a geographically distinct region of Ontario, prospective validation in an independent cohort would provide the strongest assessment of the score. 22Specifically, validation in other health care settings, including the United States, may facilitate the implementation of novel models of cancer care such as the Oncology Care Model. 8Second, additional characteristics that were unavailable such as laboratory test data and comprehensive geriatric assessments, which were predictive in other scores, [10][11][12][13][14] may further refine predictions.Third, we focused on all acute care use, rather than treatment-related or preventable visits.Previous studies suggest that 25% to 75% of acute care use while undergoing systemic therapy may be treatment related 4,48 and a similar portion may be preventable. 4,49Future research should develop and validate approaches to identify treatment-related and preventable acute care use from administrative data.Fourth, we used logistic regression to create a score that can be easily calculated and interpreted.Machine learning approaches may provide additional gains in predictive accuracy, although a recent review suggests that machine learning approaches seem to add limited value compared with logistic regression for clinical prediction models. 50

Conclusions
The PROACCT score predicts the risk of acute care use after the initiation of systemic therapy for cancer.The score quantifies an important risk of systemic therapy, which can improve the informed consent process.Future research should determine how the score affects clinical decision-making and how the score can be used to guide preventive interventions to reduce morbidity and costs.

Figure
Figure.Development and Validation Cohort CurvesROC curvesA developed a logistic regression model to predict hospitalizations in a cohort of 1579 patients receiving palliative chemotherapy at a single institution, with a C statistic of 0.71 based on internal bootstrapping.In a subsequent study, Brooks et al14 used logistic regression to predict hospitalizations among 4240 patients with stage IV or recurrent solid tumors within 3 Kaiser Permanente regional health systems.Their model included 2 variables, albumin and sodium, and had a C statistic of 0.69 in the validation cohort.
244][25]ng Acute Care Use After Initiation of Systemic Therapy for Cancer period, with no evidence of the same regimen in the prior 90 days.Only the first new regimen within the accrual period for each patient was included.Regimens were included if they were a funded, evidence-based regimen for the patient's primary cancer and indication (ie, curative or palliative) according to CCO's Systemic Treatment Funding Model.The Systemic Treatment Funding Model was developed in consultation with clinical experts and administrators, who systematically reviewed the literature to identify which treatment regimens have evidence of efficacy for each cancer type and indication.[24][25][26]Cancercenters in Ontario are reimbursed for the provision of systemic therapy only if the therapy is approved within the Systemic Treatment Funding Model.We excluded regimens outside of the Systemic Treatment Funding Model because these regimens should not be used in Ontario and are not reliably captured.More than 94% of systemic therapy treatments delivered in Ontario follow the CCO Systemic Treatment Funding Model.24Systemictherapies included intravenous and oral cytotoxic or targeted agents.Supportive regimens such as bisphosphonates and most hormonal agents were excluded, with the exception of secondline hormonal agents for prostate cancer.Regimens received by fewer than 10 patients in the development cohort were excluded.

Risk Score Development and Validation
JAMA Network Open.2019;2(10):e1912823. doi:10.1001/jamanetworkopen.2019.12823(Reprinted) October 9, 2019 3/14 Downloaded From: https://jamanetwork.com/ on 09/21/2023 characteristic and sorted into quintiles based on the observed mean number of ED visits or hospitalizations within the first 30 days after starting treatment in the development cohort (eTable in the Supplement).We combined cancer type and regimen into a single characteristic to avoid statistical issues of collinearity and complete separation.

Table 1 .
Baseline Characteristics of Patients Starting Treatment Among the 12 162 patients in the development cohort, 6903 were women and 5259 were men (mean [SD] age, 62.9[12.6]years);among the 15 845 patients in the validation cohort, 9025 were women and 6820 were men (mean [SD] age, 62.9[12.6]years).Patient and treatment characteristics were similar in both cohorts, JAMA Network Open | Oncology Predicting Acute Care Use After Initiation of Systemic Therapy for Cancer JAMA Network Open.2019;2(10):e1912823. doi:10.1001/jamanetworkopen.2019.12823(Reprinted) October 9, 2019 4/14 Downloaded From: https://jamanetwork.com/ on 09/21/2023 JAMA Network Open.2019;2(10):e1912823. doi:10.1001/jamanetworkopen.2019.12823(Reprinted) October 9, 2019 5/14 Downloaded From: https://jamanetwork.com/ on 09/21/2023 except that more patients in the validation cohort lived in rural settings and were treated in centers without the capacity for investigational drug programs or high-complexity procedures.At least 1 ED visit or hospitalization during the first 30 days after the initiation of systemic therapy occurred during 3039 treatments (25.0%) in the development cohort and 4212 treatments (26.6%) in the validation cohort.Combinations of cancer type and treatment regimen with the highest rate of AC30 included cancers with poor prognoses such as pancreatic and lung cancer, and regimens with high toxic effects such as combinations including cisplatin and docetaxel (eTable in the Supplement).In contrast, indolent cancers treated with monotherapy had lower rates of AC30.

Table 1 .
Baseline Characteristics of Patients Starting Treatment (continued) a Detailed information on cancer type is presented along with the treatment regimens in the eTable in the Supplement.bLevel 1 represents centers with investigational new

Table 2 .
Univariable and Multivariable Logistic Regression Models for Emergency Department Visit or Hospitalization Within 30 Days After Starting New Course of Systemic Therapy

Table 2 .
Univariable and Multivariable Logistic Regression Models for Emergency Department Visit or Hospitalization Within 30 Days After Starting New Course of Systemic Therapy (continued)

on 09/21/2023 symptoms
Predicting Acute Care Use After Initiation of Systemic Therapy for Cancer .The characteristics in the score are readily available to clinicians and recorded within most administrative databases, making the score calculable at the bedside or by electronic health records to provide personalized estimates of the risk of early use of acute care.These estimates can be used to improve the informed consent process and select patients for preventive interventions.Our study had a large sample size with a rich set of characteristics including treatment, demographic, comorbidity, and symptom data.Although many characteristics such as patient- Downloaded From: https://jamanetwork.com/