Customize your JAMA Network experience by selecting one or more topics from the list below.
Identify all potential conflicts of interest that might be relevant to your comment.
Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.
Err on the side of full disclosure.
If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.
Not all submitted comments are published. Please see our commenting policy for details.
Sherman SK, Poli EC, Kapadia MR, Turaga KK. Estimating Surgical Risk for Patients With Severe Comorbidities. JAMA Surg. 2018;153(8):778–780. doi:10.1001/jamasurg.2018.1055
A recent report found that surveyed residents overestimated patients’ surgical risk compared with the risk determined by the American College of Surgeons Surgical Risk Calculator (ACS-Calculator).1 However, owing to the survey’s 7 clinical scenarios describing patients in extremis with severe comorbidities, the ACS-Calculator’s estimations, which served as a reference for expected morbidity, seemed to be low. Because the ACS-Calculator was recently updated to address accuracy and calibration,2 this study sought to determine whether new ACS-Calculator estimates more closely match the residents’ estimations of risk and to determine actual outcomes of similar patients in the National Surgical Quality Improvement Project (NSQIP).
Clinical scenarios and risk estimates were as published.1 New ACS-Calculator risk estimates were obtained by entering each scenario’s risk factors and recording the results.3 The NSQIP data for this study comprised Participant User Data Files for 2012-2016 (for a total of 3 832 653 records). Morbidity included any individual complication, reoperation, or death. Records were subsetted by procedure and risk factors for each scenario, and actual morbidity and mortality were determined. The University of Iowa Institutional Review Board determined that research using the NSQIP involves only publicly available deidentified data, and therefore does not qualify as human participants research under federal regulations and is exempt from institutional review board approval.
New ACS-Calculator estimates showed markedly higher morbidity and mortality rates (Table). When risk estimates were pooled across the 7 scenarios, residents expected a mean morbidity rate of 54.4% and a mortality rate of 33.1%. The original ACS-Calculator estimates were lower, at 27.7% and 1.9%.1 With the updated ACS-Calculator, the mean estimated morbidity rate increased to 40.4% and the mean estimated mortality rate increased to 25.8%, which more closely matched the residents’ expectations.
Using actual NSQIP data, we found that including all risk factors for subsetting left few remaining patients, despite starting with 3.8 million records. Only some risk factors were therefore applied (2-4 risk factors used of 8-11 listed, depending on the scenario), and even with some negative prognostic factors excluded, the observed morbidity and mortality rates were high. For scenario 1, of 125 women in the NSQIP with hypertension and taking steroids who were undergoing an emergency total abdominal colectomy, 92.8% had complications and 51.2% died (Table). For scenario 2, of 111 men with hypertension and insulin-dependent diabetes undergoing an emergency duodenal Graham patch, 73.9% had complications and 18.9% died. The remaining scenarios returned similar findings. The mean observed morbidity rate among actual NSQIP patients across the 7 scenarios was 63.0%, and the mean observed mortality rate was 28.0%. A web tool was developed to demonstrate outcomes using different combinations of risk factors to subset NSQIP records (http://home.uchicago.edu/~sherman1/E.html).
Both the updated ACS-Calculator and actual NSQIP data find that risk in these medically complex surgical case scenarios more closely approximates the mean risk estimates reported by residents than those determined by a previous ACS-Calculator.1 Although the NSQIP outcomes in actual patients listed in the Table arbitrarily exclude some risk factors, the web tool shows that many different combinations of strong risk factors result in similarly high morbidity and mortality rates. We conclude that high rates of complications can be correctly expected for the scenario patients. Given these findings, we agree that clinicians should not be “shackled” by risk models,4 and we believe that residents should be educated to supplement, but not replace, their clinical judgment with risk estimation tools.
Risk calculators do not provide “true” reference risk, but rather projections, which vary in their applicability to individual patients. The surveyed residents correctly recognized outlier patients who, owing to their acute problems and multiple severe chronic conditions, defy attempts to mathematically estimate their course. Several authors have reported higher observed risk, as found in these scenarios, than that estimated by the ACS-Calculator and other NSQIP calculators for high-risk subgroups.5,6 Small sample sizes often explain such discrepancies,6 but clinicians most often consult the ACS-Calculator for samples of 1, their current patient. We therefore suggest that with trivial programming, the ACS-Calculator could be updated to supply actual case numbers and outcomes from the NSQIP database in addition to risk-model estimations. As our web tool demonstrates, seeing real NSQIP complication rates, and whether 10 000 or 2 NSQIP patients share the current patient’s risk factor profile, would contextualize the ACS-Calculator’s results and inform their meaning for individual patients.
Corresponding Author: Scott K. Sherman, MD, Department of Surgery, University of Chicago, 5841 S Maryland Ave, Room S214, Chicago, IL 60637 (email@example.com).
Accepted for Publication: March 4, 2018.
Published Online: May 30, 2018. doi:10.1001/jamasurg.2018.1055
Author Contributions: Drs Sherman and Turaga had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Sherman, Poli.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Sherman, Turaga.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Sherman, Poli, Turaga.
Administrative, technical, or material support: Sherman, Poli.
Study supervision: Kapadia, Turaga.
Conflict of Interest Disclosures: None reported.
Funding/Support: Dr Poli was supported by a fellowship from the Rolfe Pancreatic Cancer Foundation.
Role of the Funder/Sponsor: The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.