[Skip to Content]
[Skip to Content Landing]
Figure 1.
Cohort Selection Flowchart
Cohort Selection Flowchart

ED indicates emergency department.

Figure 2.
One-Month Survival Curves
One-Month Survival Curves

Orange line indicates patients for whom clinicians responded that they would not be surprised if the patient died in 1 month; blue line, patients for whom clinicians responded that they would be surprised if the patient died in 1 month; crosses, censored data.

Table 1.  
Cohort Demographic and Admission Characteristics
Cohort Demographic and Admission Characteristics
Table 2.  
Bivariate and Multivariable Analysis for 1-Month Mortality
Bivariate and Multivariable Analysis for 1-Month Mortality
Table 3.  
Diagnostic Test Characteristics of the Surprise Question Asked of Emergency Clinicians for the Actual 1-Month Mortalitya
Diagnostic Test Characteristics of the Surprise Question Asked of Emergency Clinicians for the Actual 1-Month Mortalitya
1.
Smith  AK, McCarthy  E, Weber  E,  et al.  Half of older Americans seen in emergency department in last month of life: most admitted to hospital, and many die there.  Health Aff (Millwood). 2012;31(6):1277-1285. doi:10.1377/hlthaff.2011.0922PubMedGoogle ScholarCrossref
2.
Wilber  ST, Blanda  M, Gerson  LW, Allen  KR.  Short-term functional decline and service use in older emergency department patients with blunt injuries.  Acad Emerg Med. 2010;17(7):679-686. doi:10.1111/j.1553-2712.2010.00799.xPubMedGoogle ScholarCrossref
3.
Deschodt  M, Devriendt  E, Sabbe  M,  et al.  Characteristics of older adults admitted to the emergency department (ED) and their risk factors for ED readmission based on comprehensive geriatric assessment: a prospective cohort study.  BMC Geriatr. 2015;15:54. doi:10.1186/s12877-015-0055-7PubMedGoogle ScholarCrossref
4.
Smith  AK, Fisher  J, Schonberg  MA,  et al.  Am I doing the right thing? provider perspectives on improving palliative care in the emergency department.  Ann Emerg Med. 2009;54(1):86-93, 93.e1. doi:10.1016/j.annemergmed.2008.08.022PubMedGoogle ScholarCrossref
5.
Oulton  J, Rhodes  SM, Howe  C, Fain  MJ, Mohler  MJ.  Advance directives for older adults in the emergency department: a systematic review.  J Palliat Med. 2015;18(6):500-505. doi:10.1089/jpm.2014.0368PubMedGoogle ScholarCrossref
6.
Steinhauser  KE, Christakis  NA, Clipp  EC, McNeilly  M, McIntyre  L, Tulsky  JA.  Factors considered important at the end of life by patients, family, physicians, and other care providers.  JAMA. 2000;284(19):2476-2482. doi:10.1001/jama.284.19.2476PubMedGoogle ScholarCrossref
7.
O’Connor  AE, Winch  S, Lukin  W, Parker  M.  Emergency medicine and futile care: taking the road less travelled.  Emerg Med Australas. 2011;23(5):640-643. doi:10.1111/j.1742-6723.2011.01435.xPubMedGoogle ScholarCrossref
8.
Ray  A, Block  SD, Friedlander  RJ, Zhang  B, Maciejewski  PK, Prigerson  HG.  Peaceful awareness in patients with advanced cancer.  J Palliat Med. 2006;9(6):1359-1368. doi:10.1089/jpm.2006.9.1359PubMedGoogle ScholarCrossref
9.
Wright  AA, Zhang  B, Ray  A,  et al.  Associations between end-of-life discussions, patient mental health, medical care near death, and caregiver bereavement adjustment.  JAMA. 2008;300(14):1665-1673. doi:10.1001/jama.300.14.1665PubMedGoogle ScholarCrossref
10.
Detering  KM, Hancock  AD, Reade  MC, Silvester  W.  The impact of advance care planning on end of life care in elderly patients: randomised controlled trial.  BMJ. 2010;340:c1345. doi:10.1136/bmj.c1345PubMedGoogle ScholarCrossref
11.
Khandelwal  N, Kross  EK, Engelberg  RA, Coe  NB, Long  AC, Curtis  JR.  Estimating the effect of palliative care interventions and advance care planning on ICU utilization: a systematic review.  Crit Care Med. 2015;43(5):1102-1111. doi:10.1097/CCM.0000000000000852PubMedGoogle ScholarCrossref
12.
Lakin  JR, Block  SD, Billings  JA,  et al.  Improving communication about serious illness in primary care: a review.  JAMA Intern Med. 2016;176(9):1380-1387. doi:10.1001/jamainternmed.2016.3212PubMedGoogle ScholarCrossref
13.
Dixon  J, Matosevic  T, Knapp  M.  The economic evidence for advance care planning: systematic review of evidence.  Palliat Med. 2015;29(10):869-884. doi:10.1177/0269216315586659PubMedGoogle ScholarCrossref
14.
Khandelwal  N, Benkeser  DC, Coe  NB, Curtis  JR.  Potential influence of advance care planning and palliative care consultation on ICU costs for patients with chronic and serious illness.  Crit Care Med. 2016;44(8):1474-1481. doi:10.1097/CCM.0000000000001675PubMedGoogle ScholarCrossref
15.
Shen  MJ, Prigerson  HG, Paulk  E,  et al.  Impact of end-of-life discussions on the reduction of Latino/non-Latino disparities in do-not-resuscitate order completion.  Cancer. 2016;122(11):1749-1756. doi:10.1002/cncr.29973PubMedGoogle ScholarCrossref
16.
Bernacki  R, Paladino  J, Neville  BA,  et al.  Effect of the Serious Illness Care Program in outpatient oncology: a cluster randomized clinical trial.  JAMA Intern Med. 2019;179(6):751-759. doi:10.1001/jamainternmed.2019.0077PubMedGoogle ScholarCrossref
17.
Paladino  J, Bernacki  R, Neville  BA,  et al.  Evaluating an intervention to improve communication between oncology clinicians and patients with life-limiting cancer: a cluster randomized clinical trial of the Serious Illness Care Program.  JAMA Oncol. 2019;5(6):801-809. doi:10.1001/jamaoncol.2019.0292PubMedGoogle ScholarCrossref
18.
Zhang  B, Wright  AA, Huskamp  HA,  et al.  Health care costs in the last week of life: associations with end-of-life conversations.  Arch Intern Med. 2009;169(5):480-488. doi:10.1001/archinternmed.2008.587PubMedGoogle ScholarCrossref
19.
Smith  TJ, Coyne  P, Cassel  B, Penberthy  L, Hopson  A, Hager  MA.  A high-volume specialist palliative care unit and team may reduce in-hospital end-of-life care costs.  J Palliat Med. 2003;6(5):699-705. doi:10.1089/109662103322515202PubMedGoogle ScholarCrossref
20.
Mack  JW, Cronin  A, Keating  NL,  et al.  Associations between end-of-life discussion characteristics and care received near death: a prospective cohort study.  J Clin Oncol. 2012;30(35):4387-4395. doi:10.1200/JCO.2012.43.6055PubMedGoogle ScholarCrossref
21.
Stone  SC, Mohanty  S, Grudzen  CR,  et al.  Emergency medicine physicians’ perspectives of providing palliative care in an emergency department.  J Palliat Med. 2011;14(12):1333-1338. doi:10.1089/jpm.2011.0106PubMedGoogle ScholarCrossref
22.
Ouchi  K, George  N, Schuur  JD,  et al.  Goals-of-care conversations for older adults with serious illness in the emergency department: challenges and opportunities.  Ann Emerg Med. 2019;74(2):276-284. doi:10.1016/j.annemergmed.2019.01.003PubMedGoogle ScholarCrossref
23.
Ouchi  K, George  N, Revette  AC,  et al.  Empower seriously ill older adults to formulate their goals for medical care in the emergency department.  J Palliat Med. 2019;22(3):267-273. doi:10.1089/jpm.2018.0360PubMedGoogle ScholarCrossref
24.
Barnes  S, Gott  M, Payne  S,  et al.  Predicting mortality among a general practice-based sample of older people with heart failure.  Chronic Illn. 2008;4(1):5-12. doi:10.1177/1742395307083783PubMedGoogle ScholarCrossref
25.
Cohen  LM, Ruthazer  R, Moss  AH, Germain  MJ.  Predicting six-month mortality for patients who are on maintenance hemodialysis.  Clin J Am Soc Nephrol. 2010;5(1):72-79. doi:10.2215/CJN.03860609PubMedGoogle ScholarCrossref
26.
Lakin  JR, Robinson  MG, Bernacki  RE,  et al.  Estimating 1-year mortality for high-risk primary care patients using the “surprise” question.  JAMA Intern Med. 2016;176(12):1863-1865. doi:10.1001/jamainternmed.2016.5928PubMedGoogle ScholarCrossref
27.
Lilley  EJ, Gemunden  SA, Kristo  G,  et al.  Utility of the “surprise” question in predicting survival among older patients with acute surgical conditions.  J Palliat Med. 2017;20(4):420-423. doi:10.1089/jpm.2016.0313PubMedGoogle ScholarCrossref
28.
Moroni  M, Zocchi  D, Bolognesi  D,  et al; SUQ-P group.  The ‘surprise’ question in advanced cancer patients: A prospective study among general practitioners.  Palliat Med. 2014;28(7):959-964. doi:10.1177/0269216314526273PubMedGoogle ScholarCrossref
29.
Moss  AH, Ganjoo  J, Sharma  S,  et al.  Utility of the “surprise” question to identify dialysis patients with high mortality.  Clin J Am Soc Nephrol. 2008;3(5):1379-1384. doi:10.2215/CJN.00940208PubMedGoogle ScholarCrossref
30.
Moss  AH, Lunney  JR, Culp  S,  et al.  Prognostic significance of the “surprise” question in cancer patients.  J Palliat Med. 2010;13(7):837-840. doi:10.1089/jpm.2010.0018PubMedGoogle ScholarCrossref
31.
Pang  WF, Kwan  BC, Chow  KM, Leung  CB, Li  PK, Szeto  CC.  Predicting 12-month mortality for peritoneal dialysis patients using the “surprise” question.  Perit Dial Int. 2013;33(1):60-66. doi:10.3747/pdi.2011.00204PubMedGoogle ScholarCrossref
32.
Ouchi  K, Jambaulikar  G, George  NR,  et al.  The “surprise question” asked of emergency physicians may predict 12-month mortality among older emergency department patients.  J Palliat Med. 2018;21(2):236-240. doi:10.1089/jpm.2017.0192PubMedGoogle ScholarCrossref
33.
Stagg  V.  CHARLSON: Stata module to calculate Charlson index of comorbidity. http://EconPapers.repec.org/RePEc:boc:bocode:s456719. Accessed June 3, 2016.
34.
Elixhauser  ASC, Palmer  L.  Clinical Classifications Software. Rockville, MD: Agency for Healthcare Research and Quality; 2015.
35.
Glare  P, Virik  K, Jones  M,  et al.  A systematic review of physicians’ survival predictions in terminally ill cancer patients.  BMJ. 2003;327(7408):195-198. doi:10.1136/bmj.327.7408.195PubMedGoogle ScholarCrossref
36.
Christakis  NA, Lamont  EB.  Extent and determinants of error in doctors’ prognoses in terminally ill patients: prospective cohort study.  BMJ. 2000;320(7233):469-472. doi:10.1136/bmj.320.7233.469PubMedGoogle ScholarCrossref
37.
Amano  K, Maeda  I, Shimoyama  S,  et al.  The accuracy of physicians’ clinical predictions of survival in patients with advanced cancer.  J Pain Symptom Manage. 2015;50(2):139-46.e1. doi:10.1016/j.jpainsymman.2015.03.004PubMedGoogle ScholarCrossref
38.
Clerkship Directors in Emergency Medicine.  Differences between the emergency department, the office, and the inpatient settings. https://www.saem.org/cdem/education/online-education/m3-curriculum/emergency-medicine-in-the-us-healthcare-system/differences-between-the-emergency-department-the-office-and-the-inpatient-setting. Accessed July 31, 2019.
39.
Nawar  EW, Niska  RW, Xu  J.  National Hospital Ambulatory Medical Care Survey: 2005 emergency department summary.  Adv Data. 2007;(386):1-32.PubMedGoogle Scholar
40.
Haydar  SA, Almeder  L, Michalakes  L, Han  PKJ, Strout  TD.  Using the surprise question to identify those with unmet palliative care needs in emergency and inpatient settings: what do clinicians think?  J Palliat Med. 2017;20(7):729-735. doi:10.1089/jpm.2016.0403PubMedGoogle ScholarCrossref
41.
Copeland-Fields  L, Griffin  T, Jenkins  T, Buckley  M, Wise  LC.  Comparison of outcome predictions made by physicians, by nurses, and by using the Mortality Prediction Model.  Am J Crit Care. 2001;10(5):313-319.PubMedGoogle Scholar
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    Emergency Medicine
    September 13, 2019

    Association of Emergency Clinicians' Assessment of Mortality Risk With Actual 1-Month Mortality Among Older Adults Admitted to the Hospital

    Author Affiliations
    • 1Department of Emergency Medicine, Brigham and Women’s Hospital, Boston, Massachusetts
    • 2Department of Emergency Medicine, Harvard Medical School, Boston, Massachusetts
    • 3Serious Illness Care Program, Ariadne Labs, Boston, Massachusetts
    • 4Department of Emergency Medicine, Maine Medical Center, Portland, Maine
    • 5Division of Sleep Medicine, Department of Medicine, Brigham and Women’s Hospital, Boston, Massachusetts
    • 6Department of Psychosocial Oncology and Palliative Care, Dana-Farber Cancer Institute, Boston, Massachusetts
    • 7Division of Palliative Medicine, Department of Medicine, Brigham and Women’s Hospital, Boston, Massachusetts
    • 8Department of Medicine, University of California, San Francisco
    • 9Department of Emergency Medicine, Alpert Medical School of Brown University, Providence, Rhode Island
    • 10Department of Medicine, Beth Israel Deaconess Medical Center, Boston, Massachusetts
    • 11Department of Psychiatry, Brigham and Women’s Hospital, Boston, Massachusetts
    JAMA Netw Open. 2019;2(9):e1911139. doi:10.1001/jamanetworkopen.2019.11139
    Key Points español 中文 (chinese)

    Question  What is the association of emergency clinicians’ assessment of mortality risk with the actual 1-month mortality among older adults who are admitted to the hospital from the emergency department?

    Findings  In this prospective cohort study including 10 737 older adults who visited the emergency department, emergency clinicians’ response of no to the question, “Would you be surprised if your patient died in the next one month?” was associated with 1-month mortality after controlling for confounders. However, the diagnostic test characteristics of the surprise question were poor.

    Meaning  Asking emergency clinicians the surprise question may be a valuable tool to identify older patients in the ED with high risk of 1-month mortality.

    Abstract

    Importance  The accuracy of mortality assessment by emergency clinicians is unknown and may affect subsequent medical decision-making.

    Objective  To determine the association of the question, “Would you be surprised if your patient died in the next one month?” (known as the surprise question) asked of emergency clinicians with actual 1-month mortality among undifferentiated older adults who visited the emergency department (ED).

    Design, Setting, and Participants  This prospective cohort study at a single academic medical center in Portland, Maine, included consecutive patients 65 years or older who received care in the ED and were subsequently admitted to the hospital from January 1, 2014, to December 31, 2015. Data analyses were conducted from January 2018 to March 2019.

    Exposures  Treating emergency clinicians were required to answer the surprise question, “Would you be surprised if your patient died in the next one month?” in the electronic medical record when placing a bed request for all patients who were being admitted to the hospital.

    Main Outcomes and Measures  The primary outcome was mortality at 1 month, assessed from the National Death Index. The secondary outcomes included accuracies of responses by both emergency clinicians and admitting internal medicine clinicians to the surprise question in identifying older patients with high 6-month and 12-month mortality.

    Results  The full cohort included 10 737 older adults (mean [SD] age, 75.9 [8.8] years; 5532 [52%] women; 10 157 [94.6%] white) in 16 223 visits treated in the ED and admitted to the hospital. There were 5132 patients (31.6%) with a Charlson Comorbidity Index score of 2 or more. Mortality rates were 8.3% at 1 month, 17.2% at 6 months, and 22.5% at 12 months. Emergency clinicians stated that they would not be surprised if the patient died in the next month for 2104 patients (19.6%). In multivariable analysis controlling for age, sex, race, admission diagnosis, and comorbid conditions, the odds of death at 1 month were higher in patients for whom clinicians answered that they would not be surprised if the patient died in the next 1 month compared with patients for whom clinicians answered that they would be surprised if the patient died in the next 1 month (odds ratio, 2.4 [95% CI, 2.2-2.7]; P < .001). However, the diagnostic test characteristics of the surprise question were poor (sensitivity, 20%; specificity, 93%; positive predictive value, 43%; negative predictive value, 82%; accuracy, 78%; area under the receiver operating curve of the multivariable model, 0.73 [95% CI, 0.72-0.74; P < .001]).

    Conclusions and Relevance  This study found that asking the surprise question of emergency clinicians may be a valuable tool to identify older patients in the ED with a high risk of 1-month mortality. The effect of implementing the surprise question to improve population-level health care for older adults in the ED who are seriously ill remains to be seen.

    Introduction

    Approximately 75% of older adults with serious, life-limiting illnesses visit the emergency department (ED) during the final 6 months of life.1 Emergency department visits often mark an inflection point in a patient’s illness trajectory, signaling a more rapid rate of decline.2,3 Many patients have not formulated their goals for care in the context of their serious illness,4 and approximately 56% to 99% of patients do not have documentation of such goals available at the time of ED presentation.5 Most patients who are seriously ill have priorities other than simply living as long as possible,6 yet they are at risk of receiving care that does not align with their goals.7

    Conversations about serious illness care goals (ie, serious illness conversation) are associated with lower rates of in-hospital death, less aggressive medical care at the end of life, earlier hospice referrals, increased peacefulness, and a 56% greater likelihood of having end-of-life wishes known and followed.8-17 Furthermore, patients with documented serious illness conversations experience a 36% reduction in the cost of end-of-life care, with an mean cost savings of $1041 per patient in the final week of life.18 A study by Smith et al19 reported that earlier serious illness conversations are 1 of 5 key changes that can reduce the costs for health care. Yet only approximately 37% of older adults who are seriously ill have these conversations with their physicians,9 and it is often late in the disease course (33 days before death20).

    Emergency physicians recognize that an ED visit provides a time and a location for older adults who are seriously ill to discuss serious illness care goals.21 For patients who are seriously ill but clinically stable and who are likely to experience a decline in illness trajectory, engaging them in such a discussion in the ED may be an ideal moment to facilitate serious illness conversations.22 Introduction or reinitiation of serious illness conversations before these patients become clinically unstable may be acceptable and feasible in the ED.23 The current clinical practice is constrained by the lack of feasible and reliable approaches to identify patients with limited life expectancy who are most likely to benefit from serious illness conversations after an ED visit. A practical method is needed to help emergency clinicians identify patients who are seriously ill and at the highest risk of mortality and to ensure that such patients receive serious illness conversations after being admitted to the hospital. The so-called surprise question, worded as “Would you be surprised if this patient died in the next 12 months?” is a method to obtain the clinician’s overall clinical impression associated with the actual 12-month mortality among patients with a life-limiting illness. The surprise question has demonstrated sensitivity of 21% to 84% and specificity of 51% to 94% in prior studies24-31 among outpatient patients with advanced cancer or chronic kidney disease undergoing hemodialysis and may be uniquely valuable in the time-pressured ED setting to identify older patients with high risk of mortality.32 The surprise question may help emergency clinicians to identify older adults who are seriously ill and who most urgently require serious illness conversation on admission so that no such patients continue the hospitalization without formulating serious illness care goals. However, to our knowledge, previous studies of the surprise question are limited by their small sample size, variable magnitudes of association, and focus on specific disease populations, limiting the generalizability of this method.

    In this large cohort study, we aimed to prospectively test the association of the surprise question with the actual 1-month mortality among a diverse population of undifferentiated older patients in the ED. We propose that emergency clinicians’ ability to estimate prognosis is the most accurate in the short term (ie, 1-month vs the traditional 12-month mortality) by the nature of their clinical practice. To most effectively devote limited resources in a time-pressured environment, the short-term prognosis for patients presenting to the ED is important to identify those who most urgently require serious illness conversations.

    Methods
    Study Design

    We conducted a prospective cohort study to examine the association of physicians’ response to the surprise question with the actual 1-month mortality of patients who visited the ED at an academic, urban hospital with an annual volume of 70 000 visits, including 24% by patients aged 65 years and older and a hospital admission rate of 47% among these older patients. The study protocol was approved by the Partners Healthcare institutional review board as human subject research. Informed consent was waived because there was no interaction with patients and their risk was strictly the potential breach of confidentiality, which was deemed a minimal risk. This study is reported following the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline. Data analyses were conducted from January 2018 to March 2019.

    Participants and Procedures

    We included all patients 65 years and older who received care in the ED and were admitted to the hospital from January 1, 2014, to December 31, 2015. We chose to include older adults who were admitted to the hospital because they would be more likely to have higher mortality compared with those who were discharged and therefore were the most appropriate patient population to study to identify older adults with the highest 1-month mortality in the ED. The patients who were discharged were more likely to live beyond 1 month, and the treating emergency clinicians were more likely to report that they would be surprised if the patient died in the next 1 month. Therefore, including the discharged patients may have risked overinflating the overall accuracy of the clinical impression. We excluded patients who were admitted to the hospital without going through the ED and those who were transferred from the ED to the cardiac catheterization laboratory or operating room after the ED (ie, expected to have high mortality).

    When placing a bed request in the electronic medical record (EMR) for any patient being admitted to the hospital, emergency clinicians were required to answer a mandatory question, “Would you be surprised if your patient died in the next one month?” Placing a bed request in the EMR is a requirement to admit a patient into the hospital from the ED. Enrollment took place consecutively 7 days per week, 24 hours per day. We obtained the death records from the National Death Index (NDI), the central computerized database containing all certified deaths in the United States, on January 1, 2018. We used the complete NDI data from 2014 to 2015 and early release data from January to December 2016, which contained more than 90% of the 2016 data. Mortality information was ascertained by matching the cohort data set from the EMR to NDI death certificate records. We used the Social Security number, first and last names, and date of birth to match the records. Participants were considered a match when all 3 variables were completely matched. Participants were assigned a vital status code (0 = assumed alive; 1 = assumed deceased) based on their status as of 1 month from the time they visited the ED. Participants with incomplete matches or multiple NDI-matched death records were excluded from the study.

    Variables

    Our primary outcome was the accuracy of clinicians’ response to the surprise question in identifying older patients in the ED with 1-month mortality. Our secondary outcomes included the accuracies of responses by both emergency clinicians and admitting internal medicine clinicians to the surprise question in identifying older patients with high 6- or 12-month mortality. We controlled for the following potential confounders in our analysis: patient demographic information (ie, age, sex, and self-reported race), Charlson Comorbidity Index score,33 source of ED arrival (eg, home, another hospital, nursing facility), and principal diagnosis using Clinical Classifications Software.34 These variables were directly extracted from the EMR as structured data variables. No manual EMR abstraction was conducted. Emergency clinicians commonly consider many clinical variables (eg, vital signs) to derive their overall clinical impressions (ie, the response to the surprise question). Some potentially confounding information may not have been considered immediately (eg, comorbid conditions); thus, we decided to include such additional variables in the multivariable model. We did not include other potential confounders easily accessible to the clinician when determining the overall clinical impression, such as vital signs, laboratory values, and other clinical scoring systems (eg, Acute Physiologic Assessment and Chronic Health Evaluation). Many, if not all, of these variables should be incorporated into clinicians’ overall clinical impression.

    Statistical Analysis

    Patients were classified into 2 subgroups based on how clinicians answered the surprise question: (1) “No, I would not be surprised if this patient died in the next one month,” and (2) “Yes, I would be surprised if this patient died in the next one month.” A generalized estimating equation model with binary outcome was used to estimate the association of death and the response to the surprise question along with other potential covariates to take into account the possible correlation between multiple visits from the same patient. We started fitting a bivariate generalized estimating equation model with a pool of potential covariates that may be clinically relevant to the outcome 1 at a time. Covariates with significant associations were included in the final multivariable model. We determined the sensitivity, specificity, positive and negative predictive values, accuracy, odds ratio (OR), and area under the receiver operating curve at 1 month using SAS statistical software version 9.4 (SAS Institute). P values were 2-tailed, and statistical significance was set at P less than .05. We presented the overall difference in 1-month survival between the 2 subgroups using Kaplan-Maier curves. Based on the study by Lilley et al,27 we expected the accuracy to be at least 0.60. To achieve the precision of at least 0.02, we hypothesized that we would require at least 10 000 patients for the sample size. We also performed a sensitivity analysis to account for intensive care unit admission status.

    Results

    We identified 19 284 ED visits by 12 517 patients from January 1, 2014, to December 31, 2015. Sixteen patients were excluded owing to unknown Social Security numbers, inconsistent Social Security number matches to the NDI records, or incomplete matches of the name, date of birth, or Social Security number. We also excluded 1764 patients because they were transferred directly to the cardiac catheterization laboratory or operating room. We identified 16 223 ED visits by 10 737 patients in our final cohort (Figure 1). The patients had a mean (SD) age of 75.9 (8.8) years and included 5532 women (51.5%), 10 157 patients (94.6%) who were white, and 5132 patients (31.6%) with a Charlson Comorbidity Index score of 2 or more (Table 1). The mortality rates were 8.3% at 1 month, 17.2% at 6 months, and 28.5% at 12 months. Eighty-five ED clinicians answered the surprise question, including 33 attending-level emergency physicians (mean time in practice, 9.2 years; 67% men), 40 resident-level emergency physicians (mean time in practice, 2.1 years; 74% men), and 12 physician assistants (mean time in practice, 9.2 years; 58% women). Within this group, the mean admission rate was 47% (range, 44%-50%) for their patients 65 years and older.

    Of 10 737 patients, the clinicians stated that they would not be surprised if the patient died in the next 1 month for 2104 patients (19.6%), and 893 of 10 737 patients (8.3%) died within 1 month. In bivariate analysis (Table 2), the odds of death at 1 month in patients for whom clinicians answered that they would not be surprised if the patient died in 1 month was 3.3-fold that of patients for whom clinicians answered that they would be surprised if the patient died in 1 month (OR, 3.3 [95% CI, 3.0-3.7]; P < .001). In multivariable analysis controlling for age, sex, race, admission diagnosis, and comorbid conditions (Table 2), the odds of death at 1 month in patients for whom clinicians answered that they would not be surprised if the patient died in 1 month was 2.4-fold that of patients for whom clinicians answered that they would be surprised if the patient died in 1 month (OR, 2.4 [95% CI, 2.2-2.7]; P < .001). The surprise question answered by clinicians demonstrated sensitivity of 20% and specificity of 93%. Given the 1-month mortality rate of 8.3% in our cohort, the positive and negative predictive values were 43% and 82%, respectively. The accuracy was 78% (Table 3). The area under the receiver operating curve of the bivariate model was 0.63 (95% CI, 0.61-0.64; P < .001), and the area under the receiver operating curve of the multivariable model was 0.73 (95% CI, 0.72-0.74; P < .001). The model performance improved from 0.70 to 0.73 by adding the surprise question as a covariate. Among 2104 patients for whom clinicians answered that they would not be surprised the patient died in 1 month, only 533 patients actually died within 1 month (Figure 2). Similar test characteristics were demonstrated by the admitting hospital clinicians. Emergency clinicians’ ability to accurately identify patients who would die vs patients who would survive was consistent at 1, 6, and 12 months after the ED visit (eTable 1 and eTable 2 in the Supplement). There were no statistically significant differences in our sensitivity analysis to account for intensive care unit admission status (eTable 3 in the Supplement). We did not have any missing data for responses to the surprise question or for 1-month mortality.

    Discussion

    Emergency clinicians asserted that they would not be surprised if 1 in 5 of their older patients in the ED died within the next month. Our study confirms that the emergency clinicians’ overall clinical assessment is associated with patients’ actual mortality (OR, 2.4 [95% CI, 2.2-2.7]; P < .001) (Table 2). However, the diagnostic test characteristic of the surprise question alone was poor, which makes it a poor screening tool for identifying patients with high risk of 1-month mortality.

    Our findings are consistent with prior studies that demonstrated the association of the clinicians’ overall clinical assessment with outcome in other clinical settings.26,27,29,30,35-37 Furthermore, our study expanded on the available literature that the overall clinical impressions of emergency clinicians who have no prior knowledge or existing relationships with patients are associated with 1-month mortality among older adults whom the clinicians admit to the hospital. Even in time-pressured ED settings, the surprise question has accuracy comparable with that of previously studied clinical settings (eg, outpatient oncology and outpatient nephrology clinics).27-29 The magnitude of association in our study was much lower (OR, 2.4) compared with prior studies (OR, 3-11).27-29 This finding may be due to a larger sample size (improving the statistical precision), higher clinical acuity and needs of patients in the ED (ie, emergency clinicians’ perception of acuity may be blunted, resulting in downward bias on the magnitude of association),38,39 and lower study mortality (8.3% vs 6% to 45%24,26,29-31) compared with prior studies in other clinical settings.

    Regardless of the diagnostic test characteristics, the surprise question, with its high specificity, remains an important adjunct for identifying patients who may require serious illness conversations. In the ED, where time is of the essence, quick identification that requires less than a few seconds is valuable and feasible, and the use of the surprise question has been shown to be feasible, acceptable, and easy to use in the ED.40 Additionally, engaging treating clinicians to assign their clinical judgment to the patients’ mortality may persuade more clinicians to solicit palliative care consultation or serious illness conversations themselves. The overestimation by the surprise question may be appropriate, since two 2019 studies16,17 have shown that there is no harm in introducing serious illness conversations earlier in the disease course. The accuracy we demonstrated of the surprise question may be useful for emergency clinicians to act on their clinical intuition to ensure serious illness conversations occur on hospital admission by communicating with the admitting physicians about a patient’s likely prognosis.

    Limitations

    Our study has several limitations. The cohort was established in a single, urban, tertiary care, academic medical center with a predominantly white population, all of which may limit the generalizability of the findings to other clinical settings. However, the clinical characteristics of the patients and clinicians were likely comparable to most EDs in the United States. The small proportion of nonwhite patients in this cohort may have produced an unstable point estimate. The clinicians in this study only made a quick judgment of their patients. Sampling bias could have occurred if the surprise question was asked before clinical deterioration of the patients. Although unlikely to bias our final results in significant ways given such a small proportion of patients in this category (7 of 10 737 patients), the potential effect of excluding patients with incomplete NDI matches is unknown. We were unable to control for potential confounding by Emergency Severity Index, chief concern, length of stay, and uncommon admission diagnoses. Some of this information may have been incorporated into clinicians’ overall clinical assessment of their patients using the surprise question. We did not include other clinicians (eg, nurses) to understand how their overall assessment of the patients may be different from that of the emergency physicians and physician assistants, although we know from a 2001 study41 that the assessments of other clinicians may be comparable to those of physicians. Further study may be warranted to investigate whether other members of the clinical team in the ED can reliably answer the surprise question to improve the scalability of implementation.

    Conclusions

    This study found that asking emergency physicians and physician assistants the surprise question may be a valuable tool to identify older patients in the ED with high risk of 1-month mortality, enhancing access to appropriate serious illness conversations and palliative care services for this population. The potential effect of implementing the surprise question to improve population-level health care for older adults in the ED who are seriously ill remains to be seen.

    Back to top
    Article Information

    Accepted for Publication: July 24, 2019.

    Published: September 13, 2019. doi:10.1001/jamanetworkopen.2019.11139

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2019 Ouchi K et al. JAMA Network Open.

    Corresponding Author: Kei Ouchi, MD, MPH, Department of Emergency Medicine, Brigham and Women’s Hospital, 75 Francis St, Neville 200, Boston, MA 02125 (kouchi@partners.org).

    Author Contributions: Drs Ouchi and Baker had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Ouchi, Strout, Haydar, Sudore, Schuur, Schonberg, Block, Tulsky.

    Acquisition, analysis, or interpretation of data: Ouchi, Strout, Haydar, Baker, Wang, Bernacki, Block.

    Drafting of the manuscript: Ouchi, Strout, Wang, Sudore.

    Critical revision of the manuscript for important intellectual content: All authors.

    Statistical analysis: Ouchi, Strout, Baker, Wang.

    Obtained funding: Ouchi.

    Administrative, technical, or material support: Ouchi, Strout, Haydar, Schuur.

    Supervision: Ouchi, Strout, Bernacki, Sudore, Schonberg, Block, Tulsky.

    Conflict of Interest Disclosures: Dr Schonberg reported receiving grants from the National Cancer Institute and the National Institute on Aging (NIA) during the conduct of the study and personal fees from UpToDate outside the submitted work. Drs Sudore and Tulsky reported receiving grants from NIA. Dr Block reported receiving personal fees from UpToDate for serving as editor in chief of the palliative care section. No other disclosures were reported.

    Funding/Support: Dr Ouchi is supported by the Grants for Early Medical and Surgical Subspecialists’ Transition to Aging Research award from the National Institute on Aging (R03 AG056449), the Emergency Medicine Foundation, and the Society of Academic Emergency Medicine.

    Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    References
    1.
    Smith  AK, McCarthy  E, Weber  E,  et al.  Half of older Americans seen in emergency department in last month of life: most admitted to hospital, and many die there.  Health Aff (Millwood). 2012;31(6):1277-1285. doi:10.1377/hlthaff.2011.0922PubMedGoogle ScholarCrossref
    2.
    Wilber  ST, Blanda  M, Gerson  LW, Allen  KR.  Short-term functional decline and service use in older emergency department patients with blunt injuries.  Acad Emerg Med. 2010;17(7):679-686. doi:10.1111/j.1553-2712.2010.00799.xPubMedGoogle ScholarCrossref
    3.
    Deschodt  M, Devriendt  E, Sabbe  M,  et al.  Characteristics of older adults admitted to the emergency department (ED) and their risk factors for ED readmission based on comprehensive geriatric assessment: a prospective cohort study.  BMC Geriatr. 2015;15:54. doi:10.1186/s12877-015-0055-7PubMedGoogle ScholarCrossref
    4.
    Smith  AK, Fisher  J, Schonberg  MA,  et al.  Am I doing the right thing? provider perspectives on improving palliative care in the emergency department.  Ann Emerg Med. 2009;54(1):86-93, 93.e1. doi:10.1016/j.annemergmed.2008.08.022PubMedGoogle ScholarCrossref
    5.
    Oulton  J, Rhodes  SM, Howe  C, Fain  MJ, Mohler  MJ.  Advance directives for older adults in the emergency department: a systematic review.  J Palliat Med. 2015;18(6):500-505. doi:10.1089/jpm.2014.0368PubMedGoogle ScholarCrossref
    6.
    Steinhauser  KE, Christakis  NA, Clipp  EC, McNeilly  M, McIntyre  L, Tulsky  JA.  Factors considered important at the end of life by patients, family, physicians, and other care providers.  JAMA. 2000;284(19):2476-2482. doi:10.1001/jama.284.19.2476PubMedGoogle ScholarCrossref
    7.
    O’Connor  AE, Winch  S, Lukin  W, Parker  M.  Emergency medicine and futile care: taking the road less travelled.  Emerg Med Australas. 2011;23(5):640-643. doi:10.1111/j.1742-6723.2011.01435.xPubMedGoogle ScholarCrossref
    8.
    Ray  A, Block  SD, Friedlander  RJ, Zhang  B, Maciejewski  PK, Prigerson  HG.  Peaceful awareness in patients with advanced cancer.  J Palliat Med. 2006;9(6):1359-1368. doi:10.1089/jpm.2006.9.1359PubMedGoogle ScholarCrossref
    9.
    Wright  AA, Zhang  B, Ray  A,  et al.  Associations between end-of-life discussions, patient mental health, medical care near death, and caregiver bereavement adjustment.  JAMA. 2008;300(14):1665-1673. doi:10.1001/jama.300.14.1665PubMedGoogle ScholarCrossref
    10.
    Detering  KM, Hancock  AD, Reade  MC, Silvester  W.  The impact of advance care planning on end of life care in elderly patients: randomised controlled trial.  BMJ. 2010;340:c1345. doi:10.1136/bmj.c1345PubMedGoogle ScholarCrossref
    11.
    Khandelwal  N, Kross  EK, Engelberg  RA, Coe  NB, Long  AC, Curtis  JR.  Estimating the effect of palliative care interventions and advance care planning on ICU utilization: a systematic review.  Crit Care Med. 2015;43(5):1102-1111. doi:10.1097/CCM.0000000000000852PubMedGoogle ScholarCrossref
    12.
    Lakin  JR, Block  SD, Billings  JA,  et al.  Improving communication about serious illness in primary care: a review.  JAMA Intern Med. 2016;176(9):1380-1387. doi:10.1001/jamainternmed.2016.3212PubMedGoogle ScholarCrossref
    13.
    Dixon  J, Matosevic  T, Knapp  M.  The economic evidence for advance care planning: systematic review of evidence.  Palliat Med. 2015;29(10):869-884. doi:10.1177/0269216315586659PubMedGoogle ScholarCrossref
    14.
    Khandelwal  N, Benkeser  DC, Coe  NB, Curtis  JR.  Potential influence of advance care planning and palliative care consultation on ICU costs for patients with chronic and serious illness.  Crit Care Med. 2016;44(8):1474-1481. doi:10.1097/CCM.0000000000001675PubMedGoogle ScholarCrossref
    15.
    Shen  MJ, Prigerson  HG, Paulk  E,  et al.  Impact of end-of-life discussions on the reduction of Latino/non-Latino disparities in do-not-resuscitate order completion.  Cancer. 2016;122(11):1749-1756. doi:10.1002/cncr.29973PubMedGoogle ScholarCrossref
    16.
    Bernacki  R, Paladino  J, Neville  BA,  et al.  Effect of the Serious Illness Care Program in outpatient oncology: a cluster randomized clinical trial.  JAMA Intern Med. 2019;179(6):751-759. doi:10.1001/jamainternmed.2019.0077PubMedGoogle ScholarCrossref
    17.
    Paladino  J, Bernacki  R, Neville  BA,  et al.  Evaluating an intervention to improve communication between oncology clinicians and patients with life-limiting cancer: a cluster randomized clinical trial of the Serious Illness Care Program.  JAMA Oncol. 2019;5(6):801-809. doi:10.1001/jamaoncol.2019.0292PubMedGoogle ScholarCrossref
    18.
    Zhang  B, Wright  AA, Huskamp  HA,  et al.  Health care costs in the last week of life: associations with end-of-life conversations.  Arch Intern Med. 2009;169(5):480-488. doi:10.1001/archinternmed.2008.587PubMedGoogle ScholarCrossref
    19.
    Smith  TJ, Coyne  P, Cassel  B, Penberthy  L, Hopson  A, Hager  MA.  A high-volume specialist palliative care unit and team may reduce in-hospital end-of-life care costs.  J Palliat Med. 2003;6(5):699-705. doi:10.1089/109662103322515202PubMedGoogle ScholarCrossref
    20.
    Mack  JW, Cronin  A, Keating  NL,  et al.  Associations between end-of-life discussion characteristics and care received near death: a prospective cohort study.  J Clin Oncol. 2012;30(35):4387-4395. doi:10.1200/JCO.2012.43.6055PubMedGoogle ScholarCrossref
    21.
    Stone  SC, Mohanty  S, Grudzen  CR,  et al.  Emergency medicine physicians’ perspectives of providing palliative care in an emergency department.  J Palliat Med. 2011;14(12):1333-1338. doi:10.1089/jpm.2011.0106PubMedGoogle ScholarCrossref
    22.
    Ouchi  K, George  N, Schuur  JD,  et al.  Goals-of-care conversations for older adults with serious illness in the emergency department: challenges and opportunities.  Ann Emerg Med. 2019;74(2):276-284. doi:10.1016/j.annemergmed.2019.01.003PubMedGoogle ScholarCrossref
    23.
    Ouchi  K, George  N, Revette  AC,  et al.  Empower seriously ill older adults to formulate their goals for medical care in the emergency department.  J Palliat Med. 2019;22(3):267-273. doi:10.1089/jpm.2018.0360PubMedGoogle ScholarCrossref
    24.
    Barnes  S, Gott  M, Payne  S,  et al.  Predicting mortality among a general practice-based sample of older people with heart failure.  Chronic Illn. 2008;4(1):5-12. doi:10.1177/1742395307083783PubMedGoogle ScholarCrossref
    25.
    Cohen  LM, Ruthazer  R, Moss  AH, Germain  MJ.  Predicting six-month mortality for patients who are on maintenance hemodialysis.  Clin J Am Soc Nephrol. 2010;5(1):72-79. doi:10.2215/CJN.03860609PubMedGoogle ScholarCrossref
    26.
    Lakin  JR, Robinson  MG, Bernacki  RE,  et al.  Estimating 1-year mortality for high-risk primary care patients using the “surprise” question.  JAMA Intern Med. 2016;176(12):1863-1865. doi:10.1001/jamainternmed.2016.5928PubMedGoogle ScholarCrossref
    27.
    Lilley  EJ, Gemunden  SA, Kristo  G,  et al.  Utility of the “surprise” question in predicting survival among older patients with acute surgical conditions.  J Palliat Med. 2017;20(4):420-423. doi:10.1089/jpm.2016.0313PubMedGoogle ScholarCrossref
    28.
    Moroni  M, Zocchi  D, Bolognesi  D,  et al; SUQ-P group.  The ‘surprise’ question in advanced cancer patients: A prospective study among general practitioners.  Palliat Med. 2014;28(7):959-964. doi:10.1177/0269216314526273PubMedGoogle ScholarCrossref
    29.
    Moss  AH, Ganjoo  J, Sharma  S,  et al.  Utility of the “surprise” question to identify dialysis patients with high mortality.  Clin J Am Soc Nephrol. 2008;3(5):1379-1384. doi:10.2215/CJN.00940208PubMedGoogle ScholarCrossref
    30.
    Moss  AH, Lunney  JR, Culp  S,  et al.  Prognostic significance of the “surprise” question in cancer patients.  J Palliat Med. 2010;13(7):837-840. doi:10.1089/jpm.2010.0018PubMedGoogle ScholarCrossref
    31.
    Pang  WF, Kwan  BC, Chow  KM, Leung  CB, Li  PK, Szeto  CC.  Predicting 12-month mortality for peritoneal dialysis patients using the “surprise” question.  Perit Dial Int. 2013;33(1):60-66. doi:10.3747/pdi.2011.00204PubMedGoogle ScholarCrossref
    32.
    Ouchi  K, Jambaulikar  G, George  NR,  et al.  The “surprise question” asked of emergency physicians may predict 12-month mortality among older emergency department patients.  J Palliat Med. 2018;21(2):236-240. doi:10.1089/jpm.2017.0192PubMedGoogle ScholarCrossref
    33.
    Stagg  V.  CHARLSON: Stata module to calculate Charlson index of comorbidity. http://EconPapers.repec.org/RePEc:boc:bocode:s456719. Accessed June 3, 2016.
    34.
    Elixhauser  ASC, Palmer  L.  Clinical Classifications Software. Rockville, MD: Agency for Healthcare Research and Quality; 2015.
    35.
    Glare  P, Virik  K, Jones  M,  et al.  A systematic review of physicians’ survival predictions in terminally ill cancer patients.  BMJ. 2003;327(7408):195-198. doi:10.1136/bmj.327.7408.195PubMedGoogle ScholarCrossref
    36.
    Christakis  NA, Lamont  EB.  Extent and determinants of error in doctors’ prognoses in terminally ill patients: prospective cohort study.  BMJ. 2000;320(7233):469-472. doi:10.1136/bmj.320.7233.469PubMedGoogle ScholarCrossref
    37.
    Amano  K, Maeda  I, Shimoyama  S,  et al.  The accuracy of physicians’ clinical predictions of survival in patients with advanced cancer.  J Pain Symptom Manage. 2015;50(2):139-46.e1. doi:10.1016/j.jpainsymman.2015.03.004PubMedGoogle ScholarCrossref
    38.
    Clerkship Directors in Emergency Medicine.  Differences between the emergency department, the office, and the inpatient settings. https://www.saem.org/cdem/education/online-education/m3-curriculum/emergency-medicine-in-the-us-healthcare-system/differences-between-the-emergency-department-the-office-and-the-inpatient-setting. Accessed July 31, 2019.
    39.
    Nawar  EW, Niska  RW, Xu  J.  National Hospital Ambulatory Medical Care Survey: 2005 emergency department summary.  Adv Data. 2007;(386):1-32.PubMedGoogle Scholar
    40.
    Haydar  SA, Almeder  L, Michalakes  L, Han  PKJ, Strout  TD.  Using the surprise question to identify those with unmet palliative care needs in emergency and inpatient settings: what do clinicians think?  J Palliat Med. 2017;20(7):729-735. doi:10.1089/jpm.2016.0403PubMedGoogle ScholarCrossref
    41.
    Copeland-Fields  L, Griffin  T, Jenkins  T, Buckley  M, Wise  LC.  Comparison of outcome predictions made by physicians, by nurses, and by using the Mortality Prediction Model.  Am J Crit Care. 2001;10(5):313-319.PubMedGoogle Scholar
    ×