Estimating Surgical Risk for Patients With Severe Comorbidities | Medical Education and Training | JAMA Surgery | JAMA Network
[Skip to Navigation]
Table.  Scenarios, Risk Factors, and Outcomes
Scenarios, Risk Factors, and Outcomes
1.
Healy  JM, Davis  KA, Pei  KY.  Comparison of internal medicine and general surgery residents’ assessments of risk of postsurgical complications in surgically complex patients.  JAMA Surg. 2018;153(3):203-207.PubMedGoogle ScholarCrossref
2.
Liu  Y, Cohen  ME, Hall  BL, Ko  CY, Bilimoria  KY.  Evaluation and enhancement of calibration in the American College of Surgeons NSQIP surgical risk calculator.  J Am Coll Surg. 2016;223(2):231-239.PubMedGoogle ScholarCrossref
3.
American College of Surgeons. ACS NSQIP surgical risk calculator. https://riskcalculator.facs.org/RiskCalculator/, Accessed December 13, 2017.
4.
Aggarwal  R.  Risk, complexity, decision making, and patient care.  JAMA Surg. 2018;153(3):208.PubMedGoogle ScholarCrossref
5.
Sherman  SK, Hrabe  JE, Charlton  ME, Cromwell  JW, Byrn  JC.  Development of an improved risk calculator for complications in proctectomy.  J Gastrointest Surg. 2014;18(5):986-994.PubMedGoogle ScholarCrossref
6.
Cohen  ME, Liu  Y, Ko  CY, Hall  BL.  An examination of American College of Surgeons NSQIP surgical risk calculator accuracy.  J Am Coll Surg. 2017;224(5):787-795.e1.PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Views 1,433
    Citations 0
    Research Letter
    August 2018

    Estimating Surgical Risk for Patients With Severe Comorbidities

    Author Affiliations
    • 1Department of Surgery, University of Chicago, Chicago, Illinois
    • 2Department of Surgery, University of Iowa Carver College of Medicine, Iowa City
    JAMA Surg. 2018;153(8):778-780. doi:10.1001/jamasurg.2018.1055

    A recent report found that surveyed residents overestimated patients’ surgical risk compared with the risk determined by the American College of Surgeons Surgical Risk Calculator (ACS-Calculator).1 However, owing to the survey’s 7 clinical scenarios describing patients in extremis with severe comorbidities, the ACS-Calculator’s estimations, which served as a reference for expected morbidity, seemed to be low. Because the ACS-Calculator was recently updated to address accuracy and calibration,2 this study sought to determine whether new ACS-Calculator estimates more closely match the residents’ estimations of risk and to determine actual outcomes of similar patients in the National Surgical Quality Improvement Project (NSQIP).

    Methods

    Clinical scenarios and risk estimates were as published.1 New ACS-Calculator risk estimates were obtained by entering each scenario’s risk factors and recording the results.3 The NSQIP data for this study comprised Participant User Data Files for 2012-2016 (for a total of 3 832 653 records). Morbidity included any individual complication, reoperation, or death. Records were subsetted by procedure and risk factors for each scenario, and actual morbidity and mortality were determined. The University of Iowa Institutional Review Board determined that research using the NSQIP involves only publicly available deidentified data, and therefore does not qualify as human participants research under federal regulations and is exempt from institutional review board approval.

    Results

    New ACS-Calculator estimates showed markedly higher morbidity and mortality rates (Table). When risk estimates were pooled across the 7 scenarios, residents expected a mean morbidity rate of 54.4% and a mortality rate of 33.1%. The original ACS-Calculator estimates were lower, at 27.7% and 1.9%.1 With the updated ACS-Calculator, the mean estimated morbidity rate increased to 40.4% and the mean estimated mortality rate increased to 25.8%, which more closely matched the residents’ expectations.

    Using actual NSQIP data, we found that including all risk factors for subsetting left few remaining patients, despite starting with 3.8 million records. Only some risk factors were therefore applied (2-4 risk factors used of 8-11 listed, depending on the scenario), and even with some negative prognostic factors excluded, the observed morbidity and mortality rates were high. For scenario 1, of 125 women in the NSQIP with hypertension and taking steroids who were undergoing an emergency total abdominal colectomy, 92.8% had complications and 51.2% died (Table). For scenario 2, of 111 men with hypertension and insulin-dependent diabetes undergoing an emergency duodenal Graham patch, 73.9% had complications and 18.9% died. The remaining scenarios returned similar findings. The mean observed morbidity rate among actual NSQIP patients across the 7 scenarios was 63.0%, and the mean observed mortality rate was 28.0%. A web tool was developed to demonstrate outcomes using different combinations of risk factors to subset NSQIP records (http://home.uchicago.edu/~sherman1/E.html).

    Discussion

    Both the updated ACS-Calculator and actual NSQIP data find that risk in these medically complex surgical case scenarios more closely approximates the mean risk estimates reported by residents than those determined by a previous ACS-Calculator.1 Although the NSQIP outcomes in actual patients listed in the Table arbitrarily exclude some risk factors, the web tool shows that many different combinations of strong risk factors result in similarly high morbidity and mortality rates. We conclude that high rates of complications can be correctly expected for the scenario patients. Given these findings, we agree that clinicians should not be “shackled” by risk models,4 and we believe that residents should be educated to supplement, but not replace, their clinical judgment with risk estimation tools.

    Risk calculators do not provide “true” reference risk, but rather projections, which vary in their applicability to individual patients. The surveyed residents correctly recognized outlier patients who, owing to their acute problems and multiple severe chronic conditions, defy attempts to mathematically estimate their course. Several authors have reported higher observed risk, as found in these scenarios, than that estimated by the ACS-Calculator and other NSQIP calculators for high-risk subgroups.5,6 Small sample sizes often explain such discrepancies,6 but clinicians most often consult the ACS-Calculator for samples of 1, their current patient. We therefore suggest that with trivial programming, the ACS-Calculator could be updated to supply actual case numbers and outcomes from the NSQIP database in addition to risk-model estimations. As our web tool demonstrates, seeing real NSQIP complication rates, and whether 10 000 or 2 NSQIP patients share the current patient’s risk factor profile, would contextualize the ACS-Calculator’s results and inform their meaning for individual patients.

    Back to top
    Article Information

    Corresponding Author: Scott K. Sherman, MD, Department of Surgery, University of Chicago, 5841 S Maryland Ave, Room S214, Chicago, IL 60637 (scott.sherman@uchospitals.edu).

    Accepted for Publication: March 4, 2018.

    Published Online: May 30, 2018. doi:10.1001/jamasurg.2018.1055

    Author Contributions: Drs Sherman and Turaga had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

    Study concept and design: Sherman, Poli.

    Acquisition, analysis, or interpretation of data: All authors.

    Drafting of the manuscript: Sherman, Turaga.

    Critical revision of the manuscript for important intellectual content: All authors.

    Statistical analysis: Sherman, Poli, Turaga.

    Administrative, technical, or material support: Sherman, Poli.

    Study supervision: Kapadia, Turaga.

    Conflict of Interest Disclosures: None reported.

    Funding/Support: Dr Poli was supported by a fellowship from the Rolfe Pancreatic Cancer Foundation.

    Role of the Funder/Sponsor: The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    References
    1.
    Healy  JM, Davis  KA, Pei  KY.  Comparison of internal medicine and general surgery residents’ assessments of risk of postsurgical complications in surgically complex patients.  JAMA Surg. 2018;153(3):203-207.PubMedGoogle ScholarCrossref
    2.
    Liu  Y, Cohen  ME, Hall  BL, Ko  CY, Bilimoria  KY.  Evaluation and enhancement of calibration in the American College of Surgeons NSQIP surgical risk calculator.  J Am Coll Surg. 2016;223(2):231-239.PubMedGoogle ScholarCrossref
    3.
    American College of Surgeons. ACS NSQIP surgical risk calculator. https://riskcalculator.facs.org/RiskCalculator/, Accessed December 13, 2017.
    4.
    Aggarwal  R.  Risk, complexity, decision making, and patient care.  JAMA Surg. 2018;153(3):208.PubMedGoogle ScholarCrossref
    5.
    Sherman  SK, Hrabe  JE, Charlton  ME, Cromwell  JW, Byrn  JC.  Development of an improved risk calculator for complications in proctectomy.  J Gastrointest Surg. 2014;18(5):986-994.PubMedGoogle ScholarCrossref
    6.
    Cohen  ME, Liu  Y, Ko  CY, Hall  BL.  An examination of American College of Surgeons NSQIP surgical risk calculator accuracy.  J Am Coll Surg. 2017;224(5):787-795.e1.PubMedGoogle ScholarCrossref
    ×