[Skip to Content]
[Skip to Content Landing]
Figure.
Development and Validation Cohort Curves
Development and Validation Cohort Curves

A, Receiver operating characteristic (ROC) curves in the development (solid line) and validation (dashed line) cohorts. Derivation area under the curve (AUC) = 0.67; and validation AUC = 0.61. The dotted line is the 45° line, which corresponds to the performance of random guesses. B, Calibration plots with a locally estimated scatterplot smoothing curve and 95% CI (shaded area) in the validation cohort. The dotted line is the 45° line, which corresponds to perfect calibration. C, Decision curve in the validation cohort.

Table 1.  
Baseline Characteristics of Patients Starting Treatment
Baseline Characteristics of Patients Starting Treatment
Table 2.  
Univariable and Multivariable Logistic Regression Models for Emergency Department Visit or Hospitalization Within 30 Days After Starting New Course of Systemic Therapy
Univariable and Multivariable Logistic Regression Models for Emergency Department Visit or Hospitalization Within 30 Days After Starting New Course of Systemic Therapy
Table 3.  
Prediction of Acute Care Use During Cancer Treatment Score
Prediction of Acute Care Use During Cancer Treatment Score
Table 4.  
Predicted Probability of Acute Care Use Within 30 Days of Starting Systemic Therapy by PROACCT Score and Observed Percentages in Development and Validation Cohorts
Predicted Probability of Acute Care Use Within 30 Days of Starting Systemic Therapy by PROACCT Score and Observed Percentages in Development and Validation Cohorts
1.
Prince  RM, Atenafu  EG, Krzyzanowska  MK.  Hospitalizations during systemic therapy for metastatic lung cancer: a systematic review of real world vs clinical trial outcomes.  JAMA Oncol. 2015;1(9):1333-1339. doi:10.1001/jamaoncol.2015.3440PubMedGoogle ScholarCrossref
2.
Yabroff  KR, Lamont  EB, Mariotto  A,  et al.  Cost of care for elderly cancer patients in the United States.  J Natl Cancer Inst. 2008;100(9):630-641. doi:10.1093/jnci/djn103PubMedGoogle ScholarCrossref
3.
Phillips  C, Deal  K, Powis  ML,  et al. Does the risk of emergency department visits and hospitalizations during systemic therapy for cancer influence patients’ decisions regarding treatment? Paper presented at: ASCO Annual Meeting; June 1, 2018; Chicago, IL.
4.
Brooks  GA, Abrams  TA, Meyerhardt  JA,  et al.  Identification of potentially avoidable hospitalizations in patients with GI cancer.  J Clin Oncol. 2014;32(6):496-503. doi:10.1200/JCO.2013.52.4330PubMedGoogle ScholarCrossref
5.
Montero  AJ, Stevenson  J, Guthrie  AE,  et al.  Reducing unplanned medical oncology readmissions by improving outpatient care transitions: a process improvement project at the Cleveland Clinic.  J Oncol Pract. 2016;12(5):e594-e602. doi:10.1200/JOP.2015.007880PubMedGoogle ScholarCrossref
6.
Colligan  EM, Ewald  E, Keating  NL,  et al.  Two innovative cancer care programs have potential to reduce utilization and spending.  Med Care. 2017;55(10):873-878. doi:10.1097/MLR.0000000000000795PubMedGoogle ScholarCrossref
7.
Cancer Care Ontario. Systemic treatment provincial plan: 2014-2019. https://www.cancercareontario.ca/en/cancer-care-ontario/programs/clinical-services/systemic-treatment/provincial-plan. Accessed August 28, 2019.
8.
Centers for Medicare & Medicaid Services. First annual report from the evaluation of the oncology care model: baseline period. https://downloads.cms.gov/files/cmmi/ocm-baselinereport.pdf. Accessed August 28, 2019.
9.
Cancer Care Ontario. Guidance for the development of a provincial approach to toxicity management: summary of recommendations from the Toxicity Management Advisory Committee. https://www.cancercareontario.ca/en/content/guidance-development-provincial-approach-toxicity-management-summary-recommendations-toxicity-management-advisory-committee. Accessed August 28, 2019.
10.
Extermann  M, Boler  I, Reich  RR,  et al.  Predicting the risk of chemotherapy toxicity in older patients: the Chemotherapy Risk Assessment Scale for High-Age Patients (CRASH) score.  Cancer. 2012;118(13):3377-3386. doi:10.1002/cncr.26646PubMedGoogle ScholarCrossref
11.
Hurria  A, Togawa  K, Mohile  SG,  et al.  Predicting chemotherapy toxicity in older adults with cancer: a prospective multicenter study.  J Clin Oncol. 2011;29(25):3457-3465. doi:10.1200/JCO.2011.34.7625PubMedGoogle ScholarCrossref
12.
Lyman  GH, Kuderer  NM, Crawford  J,  et al.  Predicting individual risk of neutropenic complications in patients receiving cancer chemotherapy.  Cancer. 2011;117(9):1917-1927. doi:10.1002/cncr.25691PubMedGoogle ScholarCrossref
13.
Brooks  GA, Kansagra  AJ, Rao  SR, Weitzman  JI, Linden  EA, Jacobson  JO.  A clinical prediction model to assess risk for chemotherapy-related hospitalization in patients initiating palliative chemotherapy.  JAMA Oncol. 2015;1(4):441-447. doi:10.1001/jamaoncol.2015.0828PubMedGoogle ScholarCrossref
14.
Brooks  GA, Uno  H, Aiello Bowles  EJ,  et al.  Hospitalization risk during chemotherapy for advanced cancer: development and validation of risk stratification models using real-world data.  JCO Clin Cancer Inform. 2019;3:1-10.PubMedGoogle Scholar
15.
Barcenas  CH, Niu  J, Zhang  N,  et al.  Risk of hospitalization according to chemotherapy regimen in early-stage breast cancer.  J Clin Oncol. 2014;32(19):2010-2017. doi:10.1200/JCO.2013.49.3676PubMedGoogle ScholarCrossref
16.
Du  XL, Osborne  C, Goodwin  JS.  Population-based assessment of hospitalizations for toxicity from chemotherapy in older women with breast cancer.  J Clin Oncol. 2002;20(24):4636-4642. doi:10.1200/JCO.2002.05.088PubMedGoogle ScholarCrossref
17.
Enright  K, Grunfeld  E, Yun  L,  et al.  Population-based assessment of emergency room visits and hospitalizations among women receiving adjuvant chemotherapy for early breast cancer.  J Oncol Pract. 2015;11(2):126-132. doi:10.1200/JOP.2014.001073PubMedGoogle ScholarCrossref
18.
Sanoff  HK, Carpenter  WR, Freburger  J,  et al.  Comparison of adverse events during 5-fluorouracil versus 5-fluorouracil/oxaliplatin adjuvant chemotherapy for stage III colon cancer: a population-based analysis.  Cancer. 2012;118(17):4309-4320. doi:10.1002/cncr.27422PubMedGoogle ScholarCrossref
19.
Fessele  KL, Hayat  MJ, Atkins  RL.  Predictors of unplanned hospitalizations in patients with nonmetastatic lung cancer during chemotherapy.  Oncol Nurs Forum. 2017;44(5):E203-E212. doi:10.1188/17.ONF.E203-E212PubMedGoogle ScholarCrossref
20.
Hassett  MJ, Rao  SR, Brozovic  S,  et al.  Chemotherapy-related hospitalization among community cancer center patients.  Oncologist. 2011;16(3):378-387. doi:10.1634/theoncologist.2010-0354PubMedGoogle ScholarCrossref
21.
Goel  V, Williams  JI, Anderson  GM, Blackstien-Hirsch  P, Fooks  C, Naylor  CD.  Patterns of Healthcare in Ontario: The ICES Practice Atlas: A Summary of Studies on the Quality of Healthcare Administrative Databases in Canada. 2nd ed. Ottawa, ON: The Canadian Medical Association; 1996.
22.
Collins  GS, Reitsma  JB, Altman  DG, Moons  KG.  Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement.  Ann Intern Med. 2015;162(1):55-63. doi:10.7326/M14-0697PubMedGoogle ScholarCrossref
23.
Cancer Care Ontario. Cancer Care Ontario’s data book—2017-2018. https://ext.cancercare.on.ca/ext/databook/db1718/databook.htm. Accessed September 4, 2019.
24.
Kaizer  L, Simanovski  V, Lalonde  C, Tariq  H, Blais  I, Evans  WK.  Using data from Ontario’s episode-based funding model to assess quality of chemotherapy.  J Oncol Pract. 2016;12(10):e870-e877. doi:10.1200/JOP.2016.013656PubMedGoogle ScholarCrossref
25.
Kaizer  L, Simanovski  V, Blais  I, Lalonde  C, Evans  WK.  International efforts in health care reform: systemic treatment funding model reform in Ontario.  J Oncol Pract. 2014;10(3):190-192. doi:10.1200/JOP.2014.001389PubMedGoogle ScholarCrossref
26.
Cancer Care Ontario. Systemic Treatment–Quality-Based Procedure (ST-QBP). https://www.cancercare.on.ca/toolbox/drugformulary/stfmregimens/. Accessed April 6, 2017.
27.
Canadian Institute for Health Information. CIHI data quality study of Ontario emergency department visits for 2004-2005. https://secure.cihi.ca/free_products/vol1_nacrs_executive_summary_nov2_2007.pdf. Accessed August 28, 2019.
28.
Canadian Institute for Health Information. Data quality documentation, discharge abstract database—multi-year information. https://www.cihi.ca/en/dad_multi-year_en.pdf. Accessed August 28, 2019.
29.
Prince  RM, Powis  M, Zer  A, Atenafu  EG, Krzyzanowska  MK.  Hospitalisations and emergency department visits in cancer patients receiving systemic therapy: systematic review and meta-analysis.  Eur J Cancer Care (Engl). 2019;28(1):e12909. doi:10.1111/ecc.12909PubMedGoogle ScholarCrossref
30.
Vandenberg  T, Coakley  N, Nayler  J,  et al.  A framework for the organization and delivery of systemic treatment.  Curr Oncol. 2009;16(1):4-15. doi:10.3747/co.v16i1.297PubMedGoogle ScholarCrossref
31.
Health Analytics Branch, Ontario Ministry of Health and Long-Term Care. Health analyst’s toolkit. http://www.health.gov.on.ca/english/providers/pub/healthanalytics/health_toolkit/health_toolkit.pdf. Accessed October 26, 2018.
32.
Austin  PC, van Walraven  C, Wodchis  WP, Newman  A, Anderson  GM.  Using the Johns Hopkins Aggregated Diagnosis Groups (ADGs) to predict mortality in a general adult population cohort in Ontario, Canada.  Med Care. 2011;49(10):932-939. doi:10.1097/MLR.0b013e318215d5e2PubMedGoogle ScholarCrossref
33.
Selby  D, Cascella  A, Gardiner  K,  et al.  A single set of numerical cutpoints to define moderate and severe symptoms for the Edmonton Symptom Assessment System.  J Pain Symptom Manage. 2010;39(2):241-249. doi:10.1016/j.jpainsymman.2009.06.010PubMedGoogle ScholarCrossref
34.
Oldenmenger  WH, de Raaf  PJ, de Klerk  C, van der Rijt  CC.  Cut points on 0-10 numeric rating scales for symptoms included in the Edmonton Symptom Assessment Scale in cancer patients: a systematic review.  J Pain Symptom Manage. 2013;45(6):1083-1093. doi:10.1016/j.jpainsymman.2012.06.007PubMedGoogle ScholarCrossref
35.
Barbera  L, Seow  H, Howell  D,  et al.  Symptom burden and performance status in a population-based cohort of ambulatory cancer patients.  Cancer. 2010;116(24):5767-5776. doi:10.1002/cncr.25681PubMedGoogle ScholarCrossref
36.
Pereira  J, Green  E, Molloy  S,  et al.  Population-based standardized symptom screening: Cancer Care Ontario’s Edmonton Symptom Assessment System and performance status initiatives.  J Oncol Pract. 2014;10(3):212-214. doi:10.1200/JOP.2014.001390PubMedGoogle ScholarCrossref
37.
Sullivan  LM, Massaro  JM, D’Agostino  RB  Sr.  Presentation of multivariate data for clinical use: the Framingham Study risk score functions.  Stat Med. 2004;23(10):1631-1660. doi:10.1002/sim.1742PubMedGoogle ScholarCrossref
38.
Hosmer  DW, Lemeshow  S, Sturdivant  RX.  Applied Logistic Regression. 3rd ed. Hoboken, NJ: Wiley & Sons; 2013. doi:10.1002/9781118548387
39.
Paul  P, Pennell  ML, Lemeshow  S.  Standardizing the power of the Hosmer-Lemeshow goodness of fit test in large data sets.  Stat Med. 2013;32(1):67-80. doi:10.1002/sim.5525PubMedGoogle ScholarCrossref
40.
Vickers  AJ, Elkin  EB.  Decision curve analysis: a novel method for evaluating prediction models.  Med Decis Making. 2006;26(6):565-574. doi:10.1177/0272989X06295361PubMedGoogle ScholarCrossref
41.
Vickers  AJ, Van Calster  B, Steyerberg  EW.  Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests.  BMJ. 2016;352:i6. doi:10.1136/bmj.i6PubMedGoogle ScholarCrossref
42.
Birmingham  LE, Cochran  T, Frey  JA, Stiffler  KA, Wilber  ST.  Emergency department use and barriers to wellness: a survey of emergency department frequent users.  BMC Emerg Med. 2017;17(1):16. doi:10.1186/s12873-017-0126-5PubMedGoogle ScholarCrossref
43.
Kraaijvanger  N, van Leeuwen  H, Rijpsma  D, Edwards  M.  Motives for self-referral to the emergency department: a systematic review of the literature.  BMC Health Serv Res. 2016;16(1):685. doi:10.1186/s12913-016-1935-zPubMedGoogle ScholarCrossref
44.
Chapman  BP, Shah  M, Friedman  B, Drayer  R, Duberstein  PR, Lyness  JM.  Personality traits predict emergency department utilization over 3 years in older patients.  Am J Geriatr Psychiatry. 2009;17(6):526-535. doi:10.1097/JGP.0b013e3181a2fbb1PubMedGoogle ScholarCrossref
45.
Shah  ND, Steyerberg  EW, Kent  DM.  Big data and predictive analytics: recalibrating expectations.  JAMA. 2018;320(1):27-28. doi:10.1001/jama.2018.5602PubMedGoogle ScholarCrossref
46.
Basch  E, Deal  AM, Kris  MG,  et al.  Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial.  J Clin Oncol. 2016;34(6):557-565. doi:10.1200/JCO.2015.63.0830PubMedGoogle ScholarCrossref
47.
Mooney  KH, Beck  SL, Wong  B,  et al.  Automated home monitoring and management of patient-reported symptoms during chemotherapy: results of the symptom care at home RCT.  Cancer Med. 2017;6(3):537-546. doi:10.1002/cam4.1002PubMedGoogle ScholarCrossref
48.
Krzyzanowska  MK, Enright  K, Moineddin  R,  et al.  Can chemotherapy-related acute care visits be accurately identified in administrative data?  J Oncol Pract. 2018;14(1):e51-e58. doi:10.1200/JOP.2017.023697PubMedGoogle ScholarCrossref
49.
Yao  J, Green  B, Krzyzanowska  MK.  Identifying potentially preventable emergency department (PPED) visits among patients with cancer in Ontario.  J Clin Oncol. 2018;36(30)(suppl):25. doi:10.1200/JCO.2018.36.30_suppl.25PubMedGoogle ScholarCrossref
50.
Christodoulou  E, Ma  J, Collins  GS, Steyerberg  EW, Verbakel  JY, Van Calster  B.  A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models.  J Clin Epidemiol. 2019;110:12-22. doi:10.1016/j.jclinepi.2019.02.004PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Views 1,081
    Original Investigation
    Oncology
    October 9, 2019

    Development and Validation of a Score to Predict Acute Care Use After Initiation of Systemic Therapy for Cancer

    Author Affiliations
    • 1Division of Medical Oncology and Hematology, Princess Margaret Cancer Centre, University Health Network, Toronto, Ontario, Canada
    • 2Department of Medicine, University of Toronto, Toronto, Ontario, Canada
    • 3ICES, Toronto, Ontario, Canada
    JAMA Netw Open. 2019;2(10):e1912823. doi:10.1001/jamanetworkopen.2019.12823
    Key Points español 中文 (chinese)

    Question  Is it possible to predict which patients with cancer will require early acute care after starting systemic therapy?

    Findings  In a cohort study, using population-based administrative databases in Ontario, Canada, a score was developed and validated to predict a visit to the emergency department or a hospitalization within the first 30 days after starting systemic therapy for cancer based on cancer type, systemic therapy regimen, previous emergency department visits, and age.

    Meaning  The findings suggest that patients at high risk for use of acute care after starting systemic therapy for cancer can be identified using a simple score that is available at the point of care.

    Abstract

    Importance  Emergency department visits and hospitalizations after starting systemic therapy for cancer are frequent, undesirable, and costly. A score to quantify the risk of needing acute care can inform decision-making and facilitate the development of preventive interventions.

    Objective  To develop and validate a score to predict early use of acute care after initiating systemic therapy for cancer.

    Design, Setting, and Participants  A retrospective population-based cohort study was conducted between July 1, 2014, and June 30, 2015. Patients with cancer were eligible if they started a new systemic therapy for cancer, regardless of line of therapy. A total of 12 162 patients in Southwestern Ontario, Canada, formed the development cohort and 15 845 patients in Northeastern Ontario formed the validation cohort. Data analysis was conducted from December 1, 2016, to August 10, 2019.

    Exposures  The Prediction of Acute Care Use During Cancer Treatment (PROACCT) score was created based on logistic regression in the development cohort. Combinations of cancer type and regimens were grouped into quintiles based on risk of needing acute care. The score was assessed in the validation cohort.

    Main Outcomes and Measures  At least 1 emergency department visit or hospitalization within 30 days after starting systemic therapy for cancer identified from administrative databases.

    Results  Among the 12 162 patients in the development cohort, 6903 were women and 5259 were men (mean [SD] age, 62.9 [12.6] years); among the 15 845 patients in the validation cohort, 9025 were women and 6820 were men (mean [SD] age, 62.9 [12.6] years). Use of acute care occurred within 30 days after initiation of systemic therapy in 3039 patients (25.0%) in the development cohort and 4212 patients (26.6%) in the validation cohort. Three characteristics predicted early use of acute care and formed the PROACCT score: combination of cancer type and treatment regimen, age, and emergency department visits in the prior year (C statistic, 0.67; 95% CI, 0.66-0.69; P < .001). Other characteristics including patient-reported symptoms did not improve performance. In the validation cohort, the PROACCT score was associated with use of acute care (odds ratio per point increase, 1.22; 95% CI, 1.20-1.24; P < .001), had a C statistic of 0.61 (95% CI, 0.60-0.62; P < .001), was reasonably calibrated, and provided net benefit in decision curve analysis.

    Conclusions and Relevance  The PROACCT score predicted the risk of early use of acute care in patients starting systemic treatment for cancer and could be incorporated at the point of care to select patients for preventive interventions. Future studies should validate the PROACCT score in other settings.

    Introduction

    Use of acute care is common during systemic therapy for cancer. A systematic review estimated that almost half of patients in observational studies are hospitalized while receiving chemotherapy.1 Use of acute care is undesirable because it may reflect suboptimal control of disease-related symptoms or toxic effects of treatment and constitutes the largest component of cancer care costs.2 The risk of acute care use with cancer treatment should be included as part of the informed consent process because this information can influence treatment decisions.3 Some acute care use is preventable4 and interventions can reduce acute care use, morbidity, and costs.5,6 Reducing acute care use is a strategic priority of health care policy makers including Cancer Care Ontario7 (CCO) and the Center for Medicare and Medicaid Innovation.8 In Ontario specifically, CCO plans to develop a “standardized and comprehensive approach to managing treatment-related toxicity,”7 including “a standardized process/tool for stratifying patients by toxicity risk.”9

    Strategies to identify patients at risk for acute care use can facilitate research on prevention. Although several existing risk scores predict specific toxic effects from chemotherapy such as febrile neutropenia,10-12 prediction scores for acute care use have focused only on hospitalizations after starting palliative chemotherapy for solid tumors and have not used population-based data.13,14 To our knowledge, to date, population-based studies on acute care use during systemic treatment for cancer have focused mostly on a single cancer type or considered few risk factors and have not generated a predictive model.15-19

    In this study, we developed and validated the Prediction of Acute Care Use During Cancer Treatment (PROACCT) score to estimate the probability of acute care use within 30 days after the initiation of systemic therapy for solid and hematologic cancers using population-based administrative data. Thirty days was selected because most acute care use after initiating systemic therapy for cancer occurs during this period20 and because early use of acute care may be the most relevant to the design and evaluation of preventive interventions.

    Methods
    Cohort Creation

    The cohort consisted of patients in Ontario who initiated an anticancer systemic therapy during the accrual period between July 1, 2014, and June 30, 2015. The cohort was created by linking multiple population-based administrative and clinical databases. These databases were linked using unique, encoded identifiers and analyzed at the Institute for Clinical Evaluative Sciences. Health care professionals and hospitals in Ontario are reimbursed for services contingent on the provision of administrative data. As a result, administrative databases in Ontario capture more than 95% of health care services rendered.21 This study was conducted in accordance with the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) statement.22 The Sunnybrook Research Institute Research Ethics Board approved this study. Analyses were undertaken at the Institute for Clinical Evaluative Sciences, a prescribed entity under Ontario’s Personal Health Information Privacy Act. As such, individual consent is not required for the use of data for approved research projects.

    Patients were identified using the Activity Level Reporting database maintained by CCO.23 Patients were excluded if they were treated for a rare cancer (defined as <20 patients in the development cohort), a myeloproliferative neoplasm, or cancers commonly treated on an inpatient basis, such as acute leukemias. Patients were also excluded if their systemic therapy was started on an inpatient basis, they were concurrently being treated for multiple cancers, they were younger than 18 years, they did not have a valid Ontario Health Insurance number, or they had a missing value for age, sex, or region of residence.

    Systemic therapy treatment courses were included if they were initiated during the accrual period. Initiation of a new treatment regimen was defined as the first dose of a systemic therapy within the accrual period, with no evidence of the same regimen in the prior 90 days. Only the first new regimen within the accrual period for each patient was included. Regimens were included if they were a funded, evidence-based regimen for the patient’s primary cancer and indication (ie, curative or palliative) according to CCO’s Systemic Treatment Funding Model. The Systemic Treatment Funding Model was developed in consultation with clinical experts and administrators, who systematically reviewed the literature to identify which treatment regimens have evidence of efficacy for each cancer type and indication.24-26 Cancer centers in Ontario are reimbursed for the provision of systemic therapy only if the therapy is approved within the Systemic Treatment Funding Model. We excluded regimens outside of the Systemic Treatment Funding Model because these regimens should not be used in Ontario and are not reliably captured. More than 94% of systemic therapy treatments delivered in Ontario follow the CCO Systemic Treatment Funding Model.24 Systemic therapies included intravenous and oral cytotoxic or targeted agents. Supportive regimens such as bisphosphonates and most hormonal agents were excluded, with the exception of second-line hormonal agents for prostate cancer. Regimens received by fewer than 10 patients in the development cohort were excluded.

    Outcomes

    The primary outcome was the occurrence of at least 1 emergency department (ED) visit or hospitalization during the first 30 days after the initiation of systemic therapy (AC30), regardless of the reason or duration. Emergency department visits were ascertained from the National Ambulatory Care Reporting System database27 and hospitalizations from the Canadian Institute for Health Information Discharge Abstract Database.28

    Baseline Characteristics

    We generated a list of characteristics that had prior evidence for an association with acute care use based on a systematic review of the literature29 and data availability. Age, sex, cancer diagnosis, systemic therapy regimen, radiotherapy treatments, and facility level where care was received were identified from the Activity Level Reporting database. Cancer diagnosis was identified using International Classification of Diseases codes, which were matched to diagnostic groupings in the CCO Systemic Treatment Funding Model. Facility level ranged from 1 through 4, where 1 represents centers with investigational new drug programs, 2 represents centers that perform high-complexity procedures such as head and neck chemoradiotherapy, 3 represents centers with medical oncologists on site, and 4 represents satellite centers without medical oncologists on site.30 Postal code from the Ontario Registered Persons Database classified residence by income quintile and as rural or urban.31 Comorbidities were assessed using Aggregated Diagnosis Groups (The Johns Hopkins ACG System, Version 10),32 which incorporate information from multiple databases. Symptoms were self-reported using the Edmonton Symptom Assessment System (ESAS); scores were dichotomized, where a score of 4 or greater reflected moderate or severe symptoms.33,34 Performance status was self-reported using the Eastern Cooperative Oncology Group (ECOG) scale. The ESAS and ECOG scores came from the Symptom Management and Reporting Database.35,36 The ESAS and ECOG values included were those closest to the regimen start date, as long as they were recorded within the preceding 30 days. Emergency department visits and hospitalizations within the previous year were identified using the same approach as with AC30.

    Statistical Analysis
    Risk Score Development and Validation

    Ontario is split into 14 geographic regions called Local Health Integration Networks for health care delivery. Patients from Southwestern Ontario formed the development cohort (Local Health Integration Networks 1-6); patients from Central, Eastern, and Northern Ontario formed the validation cohort (Local Health Integration Networks 7-14). Cancer type and regimen (ie, colorectal, FOLFIRI [leucovorin, fluorouracil, and irinotecan] plus bevacizumab) were combined into a single characteristic and sorted into quintiles based on the observed mean number of ED visits or hospitalizations within the first 30 days after starting treatment in the development cohort (eTable in the Supplement). We combined cancer type and regimen into a single characteristic to avoid statistical issues of collinearity and complete separation.

    The risk score was based on a multivariable logistic regression estimated in the development cohort with AC30 as the outcome using a backward selection procedure (2-tailed P < .05 was considered significant). Aggregated Diagnosis Groups, income quintile, rural residence, facility level, the receipt of palliative radiotherapy, ESAS score, and ECOG score were excluded from the selection procedure because these characteristics may not be readily available to clinicians and had a limited association with score performance in a sensitivity analysis. Treatment courses were excluded if there was a missing value for a characteristic included in the regression (complete case analysis). Coefficients from the multivariable regression model were converted into integer “points” to create the score.37 The points for each characteristic are summed to generate an individualized score.

    The risk score was assessed in the validation cohort. The predicted probability of AC30 came from a logistic regression using the score from the development cohort. Discrimination was evaluated visually with the receiver operating characteristic curve and statistically using the C statistic.38 Calibration was evaluated visually using a locally estimated scatterplot smoothing curve comparing predicted with observed AC30 proportions.38 Statistical testing for calibration was not performed because the large sample size could make small departures in the observed from predicted values seem significant.39 Decision curve analysis assessed the potential clinical benefits of using the PROACCT score to select patients for an intervention to prevent AC30. Decision curve analysis is described in detail elsewhere.40 In brief, decision curve analysis evaluates the value of a predictive model when making a clinical decision, for instance, selecting patients for a preventive intervention. Three strategies are compared: selecting all patients for the intervention (treat all), selecting no patients (treat none), and selecting patients using the predictive model. In decision curves, the x-axis depicts the threshold probability, which is chosen by the decision-maker. In the context of this study, a decision-maker would compare the strategies at a threshold probability of 0.2 if he or she was willing to apply the intervention to 5 patients if one of them would otherwise have had an AC30 with certainty.41 The threshold probability selected by a decision-maker will depend on the relative costs and benefits of the intervention, which can include health, financial, and other considerations, so a range is provided. The y-axis depicts the net benefit of each strategy, which is expressed in terms of the value of true positives. For example, a net benefit of 0.1 is the value of applying the preventive intervention to 10% of patients, all of whom would have had an AC30 with certainty otherwise. Data analysis was conducted from December 1, 2016, to August 10, 2019. Analyses were performed in SAS, version 7.1 (SAS Institute Inc).

    Sensitivity Analysis

    We evaluated whether the addition of facility level, Aggregated Diagnosis Groups, income quintile, ESAS score, ECOG score, and receipt of radiotherapy improved the model. We assessed how the score performed at predicting each individual component of AC30 separately: ED visits that did not lead to hospitalizations, and hospitalizations. We also assessed the score in the subgroups of patients treated with palliative and nonpalliative intent.

    Results
    Cohort Description

    We identified 12 162 patients with treatment initiations in the development cohort and 15 845 patients with treatment initiations in the validation cohort (Table 1). Among the 12 162 patients in the development cohort, 6903 were women and 5259 were men (mean [SD] age, 62.9 [12.6] years); among the 15 845 patients in the validation cohort, 9025 were women and 6820 were men (mean [SD] age, 62.9 [12.6] years). Patient and treatment characteristics were similar in both cohorts, except that more patients in the validation cohort lived in rural settings and were treated in centers without the capacity for investigational drug programs or high-complexity procedures.

    At least 1 ED visit or hospitalization during the first 30 days after the initiation of systemic therapy occurred during 3039 treatments (25.0%) in the development cohort and 4212 treatments (26.6%) in the validation cohort. Combinations of cancer type and treatment regimen with the highest rate of AC30 included cancers with poor prognoses such as pancreatic and lung cancer, and regimens with high toxic effects such as combinations including cisplatin and docetaxel (eTable in the Supplement). In contrast, indolent cancers treated with monotherapy had lower rates of AC30.

    Development of the PROACCT Score

    In univariable regressions in the development cohort, AC30 was significantly associated with all characteristics except sex, rural residence, and curative-intent radiotherapy (Table 2). The C statistic of the full multivariable model in the development cohort was 0.70 (95% CI, 0.69-0.71; P < .001). In the simple multivariable model using backward selection excluding characteristics that might not be readily accessible at the bedside (income quintile, rural residence, radiotherapy, and facility level) or those with frequently missing values (ECOG score and ESAS score), the C statistic was 0.68 (95% CI, 0.66-0.69; P < .001).

    The PROACCT score was created using coefficients from the 3 characteristics in the final multivariable model in the development cohort (Table 2). The score ranged from 0 to 13 (Table 3). The characteristics within the score were available for 11 517 patients (94.7%) in the development cohort and 14 649 patients (92.5%) in the validation cohort. The score captured most of the discrimination of the model, with a C statistic of 0.67 (95% CI, 0.66-0.69; P < .001) in the development cohort (Figure, A).

    Validation of the PROACCT Score

    In the validation cohort, the PROACCT score had a C statistic of 0.61 (95% CI, 0.60-0.62; P < .001) (Figure, A). The risk of AC30 was significantly associated with the PROACCT score in the validation cohort (odds ratio per point increase, 1.22; 95% CI, 1.20-1.24; P < .001). The score was reasonably calibrated, slightly underestimating AC30 up to a score of 10, and then overestimating for scores of 12 and 13 (Figure, B and Table 4).

    Decision curve analysis for the PROACCT score in the validation cohort is displayed in Figure, C. This decision curve demonstrates that selecting patients for an intervention using the PROACCT score had an appreciable net benefit compared with the treat all and treat none strategies for threshold probabilities of approximately 0.2 to 0.35. Therefore, selecting patients for a preventive intervention for AC30 using the PROACCT score provides a net benefit if a decision-maker is willing to apply the intervention to between 3 and 5 patients to include 1 patient who would have had an AC30.

    Sensitivity Analysis

    We generated a multivariable logistic regression model with backward selection from all available characteristics, including those that might not be readily available to clinicians: Aggregated Diagnosis Groups, income quintile, rural residence, facility level, ESAS score, ECOG score, and the receipt of radiotherapy. The resulting model included the same characteristics as the PROACCT score plus rural residence, Aggregated Diagnosis Groups, and ESAS scores for drowsiness, lack of appetite, shortness of breath, anxiety, and well-being. Discrimination was only slightly improved (development cohort C statistic, 0.69). Given that this model had comparable performance but more complexity and included characteristics that may be difficult to obtain at the bedside, we used the simpler model to generate the score.

    We also assessed the performance of the PROACCT score for the separate components of AC30. For ED visits that did not result in hospitalizations, the C statistic was 0.60 (95% CI, 0.59-0.61; P < .001) in the validation cohort. For ED visits resulting in hospitalizations or direct admissions, the C statistic was 0.59 (95% CI, 0.56-0.62; P < .001) in the validation cohort. We also assessed the score in palliative treatment–intent and nonpalliative treatment–intent subgroups, where the C statistics in the validation cohorts were 0.64 (95% CI, 0.63-0.65; P < .001) for the palliative treatment–intent cohort and 0.60 (95% CI, 0.58-0.61; P < .001) for the nonpalliative treatment–intent cohort.

    Discussion

    In this study, we developed and validated the PROACCT score to predict AC30. The score was developed using administrative and clinical data from the population of Ontario that captured a rich set of predictive characteristics including the specific treatment regimen and patient-reported symptoms. The characteristics in the score are readily available to clinicians and recorded within most administrative databases, making the score calculable at the bedside or by electronic health records to provide personalized estimates of the risk of early use of acute care. These estimates can be used to improve the informed consent process and select patients for preventive interventions.

    Our study had a large sample size with a rich set of characteristics including treatment, demographic, comorbidity, and symptom data. Although many characteristics such as patient-reported symptoms had strong univariable associations with AC30, the full model including all available characteristics had a performance similar to the simpler PROACCT score based only on cancer type, treatment regimen, ED visits, and age (C statistics in the development cohort: full model, 0.70; and simple model, 0.68). Patient-reported symptoms were of limited additional value in predicting acute care use. In the validation cohort, the PROACCT score had a C statistic of 0.61. These results suggest that characteristics observed in our data may explain only a portion of acute care use, with the remainder owing to randomness; unavailable characteristics such as laboratory test data14; difficult-to-measure characteristics such as patient beliefs,42,43 self-management skills, and behaviors44; or the availability of other health system supports.5,6

    The score was reasonably calibrated in a geographically distinct validation cohort. Calibration is often underappreciated45 but is important for decision-making by patients, clinicians, and policy makers. For instance, a recent survey of patients with cancer found that information on the risk of acute care contributes to patient decision-making regarding chemotherapy.3 Moreover, the PROACCT score provided a net benefit across reasonable probability thresholds using decision curve analysis, which incorporates both discrimination and calibration.41 Decision curve analysis demonstrated that using the score to select patients for a proactive preventive intervention would be beneficial if a decision-maker was willing to apply the intervention to between 3 and 5 patients to include 1 patient who would have AC30. This range may include the cost-benefit ratios for many interventions such as proactive symptom monitoring using electronic or interactive smartphone applications.46,47

    Previous studies generated models to predict hospitalizations and toxic effects of chemotherapy based on smaller samples. Brooks et al4 developed a logistic regression model to predict hospitalizations in a cohort of 1579 patients receiving palliative chemotherapy at a single institution, with a C statistic of 0.71 based on internal bootstrapping. In a subsequent study, Brooks et al14 used logistic regression to predict hospitalizations among 4240 patients with stage IV or recurrent solid tumors within 3 Kaiser Permanente regional health systems. Their model included 2 variables, albumin and sodium, and had a C statistic of 0.69 in the validation cohort.

    Several other scores predict specific toxic effects of chemotherapy.10-12 For instance, Lyman et al12 developed a logistic regression model to predict neutropenic complications in 3760 patients from 115 centers in the United States, with a C statistic of 0.805 using a 2:1 random split for validation. To our knowledge, the PROACCT score is the first to predict both ED visits and hospitalizations in a diverse cohort of patients across all stages of disease including solid and hematologic malignant neoplasms, derived from population-based administrative data.

    Limitations

    Our study should be interpreted in the context of its potential limitations. First, while we validated the score using a geographically distinct region of Ontario, prospective validation in an independent cohort would provide the strongest assessment of the score.22 Specifically, validation in other health care settings, including the United States, may facilitate the implementation of novel models of cancer care such as the Oncology Care Model.8 Second, additional characteristics that were unavailable such as laboratory test data and comprehensive geriatric assessments, which were predictive in other scores,10-14 may further refine predictions. Third, we focused on all acute care use, rather than treatment-related or preventable visits. Previous studies suggest that 25% to 75% of acute care use while undergoing systemic therapy may be treatment related4,48 and a similar portion may be preventable.4,49 Future research should develop and validate approaches to identify treatment-related and preventable acute care use from administrative data. Fourth, we used logistic regression to create a score that can be easily calculated and interpreted. Machine learning approaches may provide additional gains in predictive accuracy, although a recent review suggests that machine learning approaches seem to add limited value compared with logistic regression for clinical prediction models.50

    Conclusions

    The PROACCT score predicts the risk of acute care use after the initiation of systemic therapy for cancer. The score quantifies an important risk of systemic therapy, which can improve the informed consent process. Future research should determine how the score affects clinical decision-making and how the score can be used to guide preventive interventions to reduce morbidity and costs.

    Back to top
    Article Information

    Accepted for Publication: August 16, 2019.

    Published: October 9, 2019. doi:10.1001/jamanetworkopen.2019.12823

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2019 Grant RC et al. JAMA Network Open.

    Corresponding Author: Monika K. Krzyzanowska, MD, MPH, Division of Medical Oncology and Hematology, Princess Margaret Cancer Centre, University Health Network, 700 University Ave, Ste OPG 7-825, Toronto, ON M5G 2M9, Canada (monika.krzyzanowska@uhn.ca).

    Author Contributions: Drs Moineddin and Krzyzanowska had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Grant, Yao, Powis, Kukreti, Krzyzanowska.

    Acquisition, analysis, or interpretation of data: Grant, Moineddin, Yao, Kukreti, Krzyzanowska.

    Drafting of the manuscript: Grant, Krzyzanowska.

    Critical revision of the manuscript for important intellectual content: All authors.

    Statistical analysis: Moineddin.

    Obtained funding: Krzyzanowska.

    Administrative, technical, or material support: Grant, Powis, Krzyzanowska.

    Supervision: Kukreti, Krzyzanowska.

    Conflict of Interest Disclosures: Dr Kukreti reported receiving honoraria from Celgene, Amgen, and Takeda during the conduct of the study. Dr Krzyzanowska reported receiving grants from Ontario Institute for Cancer Research during the conduct of the study; receiving funding to the University Health Network from Eisai and Exelixis; and receiving personal fees from Eisai outside the submitted work. No other disclosures were reported.

    Funding/Support: Funding for this project was provided by the Ontario Institute for Cancer Research Health Services Research program. This study was supported by ICES, which is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care.

    Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Disclaimer: The opinions, results, and conclusions reported in this article are those of the authors and are independent from the funding sources. No endorsement by ICES or the Ontario Ministry of Health and Long-Term Care is intended or should be inferred.

    References
    1.
    Prince  RM, Atenafu  EG, Krzyzanowska  MK.  Hospitalizations during systemic therapy for metastatic lung cancer: a systematic review of real world vs clinical trial outcomes.  JAMA Oncol. 2015;1(9):1333-1339. doi:10.1001/jamaoncol.2015.3440PubMedGoogle ScholarCrossref
    2.
    Yabroff  KR, Lamont  EB, Mariotto  A,  et al.  Cost of care for elderly cancer patients in the United States.  J Natl Cancer Inst. 2008;100(9):630-641. doi:10.1093/jnci/djn103PubMedGoogle ScholarCrossref
    3.
    Phillips  C, Deal  K, Powis  ML,  et al. Does the risk of emergency department visits and hospitalizations during systemic therapy for cancer influence patients’ decisions regarding treatment? Paper presented at: ASCO Annual Meeting; June 1, 2018; Chicago, IL.
    4.
    Brooks  GA, Abrams  TA, Meyerhardt  JA,  et al.  Identification of potentially avoidable hospitalizations in patients with GI cancer.  J Clin Oncol. 2014;32(6):496-503. doi:10.1200/JCO.2013.52.4330PubMedGoogle ScholarCrossref
    5.
    Montero  AJ, Stevenson  J, Guthrie  AE,  et al.  Reducing unplanned medical oncology readmissions by improving outpatient care transitions: a process improvement project at the Cleveland Clinic.  J Oncol Pract. 2016;12(5):e594-e602. doi:10.1200/JOP.2015.007880PubMedGoogle ScholarCrossref
    6.
    Colligan  EM, Ewald  E, Keating  NL,  et al.  Two innovative cancer care programs have potential to reduce utilization and spending.  Med Care. 2017;55(10):873-878. doi:10.1097/MLR.0000000000000795PubMedGoogle ScholarCrossref
    7.
    Cancer Care Ontario. Systemic treatment provincial plan: 2014-2019. https://www.cancercareontario.ca/en/cancer-care-ontario/programs/clinical-services/systemic-treatment/provincial-plan. Accessed August 28, 2019.
    8.
    Centers for Medicare & Medicaid Services. First annual report from the evaluation of the oncology care model: baseline period. https://downloads.cms.gov/files/cmmi/ocm-baselinereport.pdf. Accessed August 28, 2019.
    9.
    Cancer Care Ontario. Guidance for the development of a provincial approach to toxicity management: summary of recommendations from the Toxicity Management Advisory Committee. https://www.cancercareontario.ca/en/content/guidance-development-provincial-approach-toxicity-management-summary-recommendations-toxicity-management-advisory-committee. Accessed August 28, 2019.
    10.
    Extermann  M, Boler  I, Reich  RR,  et al.  Predicting the risk of chemotherapy toxicity in older patients: the Chemotherapy Risk Assessment Scale for High-Age Patients (CRASH) score.  Cancer. 2012;118(13):3377-3386. doi:10.1002/cncr.26646PubMedGoogle ScholarCrossref
    11.
    Hurria  A, Togawa  K, Mohile  SG,  et al.  Predicting chemotherapy toxicity in older adults with cancer: a prospective multicenter study.  J Clin Oncol. 2011;29(25):3457-3465. doi:10.1200/JCO.2011.34.7625PubMedGoogle ScholarCrossref
    12.
    Lyman  GH, Kuderer  NM, Crawford  J,  et al.  Predicting individual risk of neutropenic complications in patients receiving cancer chemotherapy.  Cancer. 2011;117(9):1917-1927. doi:10.1002/cncr.25691PubMedGoogle ScholarCrossref
    13.
    Brooks  GA, Kansagra  AJ, Rao  SR, Weitzman  JI, Linden  EA, Jacobson  JO.  A clinical prediction model to assess risk for chemotherapy-related hospitalization in patients initiating palliative chemotherapy.  JAMA Oncol. 2015;1(4):441-447. doi:10.1001/jamaoncol.2015.0828PubMedGoogle ScholarCrossref
    14.
    Brooks  GA, Uno  H, Aiello Bowles  EJ,  et al.  Hospitalization risk during chemotherapy for advanced cancer: development and validation of risk stratification models using real-world data.  JCO Clin Cancer Inform. 2019;3:1-10.PubMedGoogle Scholar
    15.
    Barcenas  CH, Niu  J, Zhang  N,  et al.  Risk of hospitalization according to chemotherapy regimen in early-stage breast cancer.  J Clin Oncol. 2014;32(19):2010-2017. doi:10.1200/JCO.2013.49.3676PubMedGoogle ScholarCrossref
    16.
    Du  XL, Osborne  C, Goodwin  JS.  Population-based assessment of hospitalizations for toxicity from chemotherapy in older women with breast cancer.  J Clin Oncol. 2002;20(24):4636-4642. doi:10.1200/JCO.2002.05.088PubMedGoogle ScholarCrossref
    17.
    Enright  K, Grunfeld  E, Yun  L,  et al.  Population-based assessment of emergency room visits and hospitalizations among women receiving adjuvant chemotherapy for early breast cancer.  J Oncol Pract. 2015;11(2):126-132. doi:10.1200/JOP.2014.001073PubMedGoogle ScholarCrossref
    18.
    Sanoff  HK, Carpenter  WR, Freburger  J,  et al.  Comparison of adverse events during 5-fluorouracil versus 5-fluorouracil/oxaliplatin adjuvant chemotherapy for stage III colon cancer: a population-based analysis.  Cancer. 2012;118(17):4309-4320. doi:10.1002/cncr.27422PubMedGoogle ScholarCrossref
    19.
    Fessele  KL, Hayat  MJ, Atkins  RL.  Predictors of unplanned hospitalizations in patients with nonmetastatic lung cancer during chemotherapy.  Oncol Nurs Forum. 2017;44(5):E203-E212. doi:10.1188/17.ONF.E203-E212PubMedGoogle ScholarCrossref
    20.
    Hassett  MJ, Rao  SR, Brozovic  S,  et al.  Chemotherapy-related hospitalization among community cancer center patients.  Oncologist. 2011;16(3):378-387. doi:10.1634/theoncologist.2010-0354PubMedGoogle ScholarCrossref
    21.
    Goel  V, Williams  JI, Anderson  GM, Blackstien-Hirsch  P, Fooks  C, Naylor  CD.  Patterns of Healthcare in Ontario: The ICES Practice Atlas: A Summary of Studies on the Quality of Healthcare Administrative Databases in Canada. 2nd ed. Ottawa, ON: The Canadian Medical Association; 1996.
    22.
    Collins  GS, Reitsma  JB, Altman  DG, Moons  KG.  Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement.  Ann Intern Med. 2015;162(1):55-63. doi:10.7326/M14-0697PubMedGoogle ScholarCrossref
    23.
    Cancer Care Ontario. Cancer Care Ontario’s data book—2017-2018. https://ext.cancercare.on.ca/ext/databook/db1718/databook.htm. Accessed September 4, 2019.
    24.
    Kaizer  L, Simanovski  V, Lalonde  C, Tariq  H, Blais  I, Evans  WK.  Using data from Ontario’s episode-based funding model to assess quality of chemotherapy.  J Oncol Pract. 2016;12(10):e870-e877. doi:10.1200/JOP.2016.013656PubMedGoogle ScholarCrossref
    25.
    Kaizer  L, Simanovski  V, Blais  I, Lalonde  C, Evans  WK.  International efforts in health care reform: systemic treatment funding model reform in Ontario.  J Oncol Pract. 2014;10(3):190-192. doi:10.1200/JOP.2014.001389PubMedGoogle ScholarCrossref
    26.
    Cancer Care Ontario. Systemic Treatment–Quality-Based Procedure (ST-QBP). https://www.cancercare.on.ca/toolbox/drugformulary/stfmregimens/. Accessed April 6, 2017.
    27.
    Canadian Institute for Health Information. CIHI data quality study of Ontario emergency department visits for 2004-2005. https://secure.cihi.ca/free_products/vol1_nacrs_executive_summary_nov2_2007.pdf. Accessed August 28, 2019.
    28.
    Canadian Institute for Health Information. Data quality documentation, discharge abstract database—multi-year information. https://www.cihi.ca/en/dad_multi-year_en.pdf. Accessed August 28, 2019.
    29.
    Prince  RM, Powis  M, Zer  A, Atenafu  EG, Krzyzanowska  MK.  Hospitalisations and emergency department visits in cancer patients receiving systemic therapy: systematic review and meta-analysis.  Eur J Cancer Care (Engl). 2019;28(1):e12909. doi:10.1111/ecc.12909PubMedGoogle ScholarCrossref
    30.
    Vandenberg  T, Coakley  N, Nayler  J,  et al.  A framework for the organization and delivery of systemic treatment.  Curr Oncol. 2009;16(1):4-15. doi:10.3747/co.v16i1.297PubMedGoogle ScholarCrossref
    31.
    Health Analytics Branch, Ontario Ministry of Health and Long-Term Care. Health analyst’s toolkit. http://www.health.gov.on.ca/english/providers/pub/healthanalytics/health_toolkit/health_toolkit.pdf. Accessed October 26, 2018.
    32.
    Austin  PC, van Walraven  C, Wodchis  WP, Newman  A, Anderson  GM.  Using the Johns Hopkins Aggregated Diagnosis Groups (ADGs) to predict mortality in a general adult population cohort in Ontario, Canada.  Med Care. 2011;49(10):932-939. doi:10.1097/MLR.0b013e318215d5e2PubMedGoogle ScholarCrossref
    33.
    Selby  D, Cascella  A, Gardiner  K,  et al.  A single set of numerical cutpoints to define moderate and severe symptoms for the Edmonton Symptom Assessment System.  J Pain Symptom Manage. 2010;39(2):241-249. doi:10.1016/j.jpainsymman.2009.06.010PubMedGoogle ScholarCrossref
    34.
    Oldenmenger  WH, de Raaf  PJ, de Klerk  C, van der Rijt  CC.  Cut points on 0-10 numeric rating scales for symptoms included in the Edmonton Symptom Assessment Scale in cancer patients: a systematic review.  J Pain Symptom Manage. 2013;45(6):1083-1093. doi:10.1016/j.jpainsymman.2012.06.007PubMedGoogle ScholarCrossref
    35.
    Barbera  L, Seow  H, Howell  D,  et al.  Symptom burden and performance status in a population-based cohort of ambulatory cancer patients.  Cancer. 2010;116(24):5767-5776. doi:10.1002/cncr.25681PubMedGoogle ScholarCrossref
    36.
    Pereira  J, Green  E, Molloy  S,  et al.  Population-based standardized symptom screening: Cancer Care Ontario’s Edmonton Symptom Assessment System and performance status initiatives.  J Oncol Pract. 2014;10(3):212-214. doi:10.1200/JOP.2014.001390PubMedGoogle ScholarCrossref
    37.
    Sullivan  LM, Massaro  JM, D’Agostino  RB  Sr.  Presentation of multivariate data for clinical use: the Framingham Study risk score functions.  Stat Med. 2004;23(10):1631-1660. doi:10.1002/sim.1742PubMedGoogle ScholarCrossref
    38.
    Hosmer  DW, Lemeshow  S, Sturdivant  RX.  Applied Logistic Regression. 3rd ed. Hoboken, NJ: Wiley & Sons; 2013. doi:10.1002/9781118548387
    39.
    Paul  P, Pennell  ML, Lemeshow  S.  Standardizing the power of the Hosmer-Lemeshow goodness of fit test in large data sets.  Stat Med. 2013;32(1):67-80. doi:10.1002/sim.5525PubMedGoogle ScholarCrossref
    40.
    Vickers  AJ, Elkin  EB.  Decision curve analysis: a novel method for evaluating prediction models.  Med Decis Making. 2006;26(6):565-574. doi:10.1177/0272989X06295361PubMedGoogle ScholarCrossref
    41.
    Vickers  AJ, Van Calster  B, Steyerberg  EW.  Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests.  BMJ. 2016;352:i6. doi:10.1136/bmj.i6PubMedGoogle ScholarCrossref
    42.
    Birmingham  LE, Cochran  T, Frey  JA, Stiffler  KA, Wilber  ST.  Emergency department use and barriers to wellness: a survey of emergency department frequent users.  BMC Emerg Med. 2017;17(1):16. doi:10.1186/s12873-017-0126-5PubMedGoogle ScholarCrossref
    43.
    Kraaijvanger  N, van Leeuwen  H, Rijpsma  D, Edwards  M.  Motives for self-referral to the emergency department: a systematic review of the literature.  BMC Health Serv Res. 2016;16(1):685. doi:10.1186/s12913-016-1935-zPubMedGoogle ScholarCrossref
    44.
    Chapman  BP, Shah  M, Friedman  B, Drayer  R, Duberstein  PR, Lyness  JM.  Personality traits predict emergency department utilization over 3 years in older patients.  Am J Geriatr Psychiatry. 2009;17(6):526-535. doi:10.1097/JGP.0b013e3181a2fbb1PubMedGoogle ScholarCrossref
    45.
    Shah  ND, Steyerberg  EW, Kent  DM.  Big data and predictive analytics: recalibrating expectations.  JAMA. 2018;320(1):27-28. doi:10.1001/jama.2018.5602PubMedGoogle ScholarCrossref
    46.
    Basch  E, Deal  AM, Kris  MG,  et al.  Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial.  J Clin Oncol. 2016;34(6):557-565. doi:10.1200/JCO.2015.63.0830PubMedGoogle ScholarCrossref
    47.
    Mooney  KH, Beck  SL, Wong  B,  et al.  Automated home monitoring and management of patient-reported symptoms during chemotherapy: results of the symptom care at home RCT.  Cancer Med. 2017;6(3):537-546. doi:10.1002/cam4.1002PubMedGoogle ScholarCrossref
    48.
    Krzyzanowska  MK, Enright  K, Moineddin  R,  et al.  Can chemotherapy-related acute care visits be accurately identified in administrative data?  J Oncol Pract. 2018;14(1):e51-e58. doi:10.1200/JOP.2017.023697PubMedGoogle ScholarCrossref
    49.
    Yao  J, Green  B, Krzyzanowska  MK.  Identifying potentially preventable emergency department (PPED) visits among patients with cancer in Ontario.  J Clin Oncol. 2018;36(30)(suppl):25. doi:10.1200/JCO.2018.36.30_suppl.25PubMedGoogle ScholarCrossref
    50.
    Christodoulou  E, Ma  J, Collins  GS, Steyerberg  EW, Verbakel  JY, Van Calster  B.  A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models.  J Clin Epidemiol. 2019;110:12-22. doi:10.1016/j.jclinepi.2019.02.004PubMedGoogle ScholarCrossref
    ×