The degree of overestimation by both groups (in absolute percentage) was compared with the NSQIP estimates of risk for the scenarios. The internal medical and general surgery trainees overestimated risk in every category by a mean of 30%. Error bars indicate SD.
eFigure 1. Scenario 1 Representative of a Diverse Emergency General Surgery Practice in Complex Patients Likely to Be Comanaged by Medical and Surgical Services
eFigure 2. Scenario 2 Representative of a Diverse Emergency General Surgery Practice in Complex Patients Likely to Be Comanaged by Medical and Surgical Services
eFigure 3. Scenario 3 Representative of a Diverse Emergency General Surgery Practice in Complex Patients Likely to Be Comanaged by Medical and Surgical Services
eFigure 4. Scenario 4 Representative of a Diverse Emergency General Surgery Practice in Complex Patients Likely to Be Comanaged by Medical and Surgical Services
eFigure 5. Scenario 5 Representative of a Diverse Emergency General Surgery Practice in Complex Patients Likely to Be Comanaged by Medical and Surgical Services
eFigure 6. Scenario 6 Representative of a Diverse Emergency General Surgery Practice in Complex Patients Likely to Be Comanaged by Medical and Surgical Services
eFigure 7. Scenario 7 Representative of a Diverse Emergency General Surgery Practice in Complex Patients Likely to Be Comanaged by Medical and Surgical Services
eFigure 8. Subgroup Analysis: Internal Medicine Senior Residents Significantly Overestimate Risk Compared to Internal Medicine Interns
eFigure 9. Subgroup Analysis: Surgery Senior and Junior Residents Have Similar Estimates of Surgical Risk
eFigure 10. Subgroup Analysis: Internal Medicine and Surgery Senior Residents Have Similar Estimates of Surgical Risk
Healy JM, Davis KA, Pei KY. Comparison of Internal Medicine and General Surgery Residents' Assessments of Risk of Postsurgical Complications in Surgically Complex Patients. JAMA Surg. Published online October 11, 2017. doi:10.1001/jamasurg.2017.3936
How do internal medicine and general surgery trainees compare in their assessment of risk of postsurgical complications in surgically complex patients?
In this study, assessment of risk of postsurgical complications was not significantly different between internal medicine and general surgery trainee groups, but both groups consistently overestimated risk compared with the American College of Surgeons’ National Surgical Quality Improvement Project risk-adjusted model.
Trainees may overestimate risks when counseling patients; a broad range of estimates implies a lack of consensus on true risk for surgically complex patients.
Anticipating postsurgical complications is a vital physician skill, particularly when counseling surgically complex patients on their risks of intervention. Although internists and surgeons both counsel patients on surgical risks, it is uncertain who is better equipped to accurately anticipate surgical complications.
To examine how internal medicine and general surgery trainees compare in their assessment of risk of surgically complex patients.
Design, Setting, and Participants
General surgery and internal medicine residents (urban, tertiary, and academic medical center) answered an anonymous, online assessment of 7 real-life, complex clinical scenarios. Participants estimated the chance of any morbidity, mortality, surgical site infection, pneumonia, and cardiac complications. Scenarios represented a diverse general surgery practice, including colectomy, duodenal ulcer repair, inguinal hernia repair, perforated viscus exploration, small-bowel resection, cholecystectomy, and mastectomy in surgically complex patients likely to be comanaged by surgical and internal medicine services.
Main Outcomes and Measures
Responses were compared with risk-adjusted outcomes reported by the American College of Surgeons’ National Surgical Quality Improvement Project (NSQIP) online calculator.
A total of 76 general surgery residents (50 [65.8%] male and 26 [34.2%] female) and 76 internal medicine residents (36 [47.4%] male and 40 [52.6%] female) participated (64% overall response rate). General surgery residents were significantly more confident with their responses (general surgery residents’ mean response, 3.6 [95% CI, 3.4-2.8]; internal medicine residents’ mean response, 2.8 [95% CI, 2.6-3.0]; P < .001) and with not offering operations (general surgery residents’ mean response, 4.3 [95% CI, 4.1-4.4]; internal medicine residents’ mean response, 3.7 [95% CI, 3.4-3.9]; P = .006) but less likely to discuss code status (general surgery residents’ mean response, 3.2 [95% CI, 2.9-3.4]; internal medicine residents’ mean response, 3.8 [95% CI, 3.5-4.1]; P < .001) or consult risk-adjusted models, such as NSQIP (general surgery residents’ mean response, 2.9 [95% CI, 2.7-3.1]; internal medicine residents’ mean response, 3.7 [95% CI, 3.4-4.0]; P < .001). For 91% of clinical estimates, both groups similarly overestimated every type of risk; in 9% of estimates, internal medicine residents had higher overestimates. Estimates varied significantly, with wide 95% CIs; however, only 11% of the NSQIP estimates fell within the 95% CIs. Overall, the mean percentages of the estimates ranged from 26% to 33% over NSQIP estimates for all complications.
Conclusions and Relevance
General surgery and internal medicine residents demonstrated similar estimates of postoperative complications and death. Both groups overestimated risks in surgically complex patient scenarios compared with NSQIP risk calculator estimates. This near-universal overestimation of risk underscores the importance of developing risk-estimation resources for internists and surgeons.
Anticipating postsurgical complications is a vital skill and a shared responsibility of general surgeons and internal medicine physicians, particularly for surgically complex patients. Most physicians develop their sense of operative risk throughout training, quality improvement conferences, in-training and board examinations, and review of literature. Senior surgeons often develop a personal database of outcomes of surgically complex patients over time, but these outcomes are difficult to generalize. For patients with complex cardiopulmonary disease or medical comorbidities, surgeons may seek comanagement with internal medical physicians and often rely on these colleagues to cocounsel patients on expected postoperative outcomes.
Although internists and surgeons counsel patients on surgical risks, it is unclear which group is better equipped to accurately determine surgical complications. Anticipating risk in medicine has always been challenging and is subject to the multiple biases of medical practice, such as the availability heuristic and a strong tendency toward anecdotal recall.1- 3 Recognition of these predispositions has prompted a vast amount of literature on methods to objectify and calculate preoperative cardiac risk, but few studies4- 6 take into account multisystem complexity and the intersection of multiple disease states. In addition, patients are living longer, and their increased complexity attributable to underlying comorbidities significantly influences their operative risk.7 Recent evidence suggests that internal medicine physicians often overestimate risks and benefits of treatment when faced with complex patients.8 This information is sparse for surgeons, although one study9 found that surgery residents and attending surgeons similarly and significantly overestimated postoperative risks. Our current study assessed the accuracy of estimates based on real-life case scenarios while comparing estimates between internal medicine and general surgery residents.
To enact an objective, risk-adjusted model to assist surgeons in anticipating postoperative complications, the American College of Surgeons’ National Surgical Quality Improvement Program (NSQIP) produced an online surgical risk calculator. This web-based calculator allows users to enter 20 patient characteristics associated with risk and is designed to synthesize outcomes data from the 2.7 million records in the NSQIP national database to estimate the 30-day risk of specific complications. The output of this calculator suggests the possible risk of 12 specific types of complications, including morbidity, mortality, cardiac and pulmonary complications, urinary tract infections, venous thromboembolism, renal failure, readmission, additional operations, and need for discharge to a nursing or rehabilitation facility. Although the validity and accuracy of the results of this tool have been the subject of debate, the calculator persists as one of the few available risk-adjusted models for surgical outcomes available for clinicians. We hypothesized that internal medicine and general surgical trainees significantly overestimate surgical risks for surgically complex patients compared with the NSQIP risk calculator with similar accuracy using NSQIP as the standard.
Residents at an urban, tertiary, academic medical center were invited to participate in an online assessment. Seven complex, real-life clinical scenarios were presented to participants via anonymous, online modules. For each scenario, participants estimated the likelihood of morbidity, mortality, surgical site infection, pneumonia, and cardiac complications on a scale rating probability of outcome of interest from 0% (not probable) to 100% (highly probable). Scenarios were representative of a diverse emergency general surgery practice and included colectomy, duodenal ulcer repair, inguinal hernia repair, perforated viscus exploration, small-bowel resection, cholecystectomy, and mastectomy in surgically complex patients likely to be comanaged by medical and surgical services (eFigures 1-7 in the Supplement). Participant responses were compared with risk-adjusted outcome measures using the NSQIP online calculator. Written informed consent was obtained from the participants, and all data were deidentified. This study was approved by the Human Investigation Committee, General Surgery and Internal Medicine, Yale University.
All data and statistical analysis were performed using SPSS statistical software, version 22.0 (IBM Inc). Figures were created using GraphPad Prism, version 7.0 (GraphPad Software Inc). A priori power analysis was performed, and a sample size of 150 (75 in each group) was determined to be necessary to achieve 80% power to detect a 10% difference between groups with a 2-sided 5% significance level. Normality of continuous variables was assessed using the Kolmogorov-Smirnov test to determine whether parametric or nonparametric statistical tests should be used. To determine whether medicine and surgical residents had statistically significant differences between their assessments of risk for the individual questions, independent-sample, 2-tailed, unpaired t tests or Mann-Whitney nonparametric tests were used as indicated. Differences in participant responses and NSQIP estimates were reported as absolute percentage differences of the mean. In addition, subgroup analyses were performed to compare junior vs senior residents within each specialty and internal medicine vs general surgery senior residents. Junior residents are defined as postgraduate year (PGY) 1 for internal medicine and less than PGY3 for general surgery. Senior residents are defined as PGY2 to PGY4 for internal medicine (chief residents were included as PGY4) and PGY3 to PGY5 for general surgery.
Of the invited participants, 76 general surgery residents (50 [65.8%] male and 26 [34.2%] female) and 76 internal medicine residents (36 [47.4%] male and 40 [52.6%] female) participated (64% overall response rate). The overall demographic characteristics of the participants are detailed in Table 1. Background questions regarding confidence with risk assessment, comfort with risk discussions, and routine use of risk-adjusted models for prediction of complications were assessed on a 5-point Likert scale, with 1 indicating strongly disagree and 5 indicating strongly agree (Table 2). General surgery residents were significantly more confident with their responses (general surgery residents’ mean response, 3.6 [95% CI, 3.4-3.8]; internal medicine residents’ mean response, 2.8 [95% CI, 2.6-3.0]; P < .001) and with not offering operations (general surgery residents’ mean response, 4.3 [95% CI, 4.1-4.4]; internal medicine residents’ mean response, 3.7 [95% CI, 3.4-3.9]; P < .001) but less likely to discuss code status (general surgery residents’ mean response, 3.2 [95% CI, 2.9-3.4]; internal medicine residents’ mean response, 3.8 [95% CI, 3.5-4.1]; P < .001) or consult risk-adjusted models, such as NSQIP (general surgery residents’ mean response, 2.9 [95% CI, 2.7-3.1]; internal medicine residents’ mean response, 3.7 [95% CI, 3.4-4.0]; P < .001).
For 91% of clinical estimates, both groups similarly overestimated every type of risk. However, internal medicine residents had significantly higher overestimates in 9% of the scenario predictions (P < .05). For both groups, estimates varied significantly, with wide 95% CIs; only 11% of NSQIP estimates fell within the 95% CIs. All scenarios and results are detailed in eFigures 1-7 in the Supplement. Overall, the mean percentages of the estimates ranged from 26% to 33% over NSQIP estimates for all complications (Figure).
In the subgroup analysis, internal medicine junior residents’ estimates of risk were significantly lower than their senior collegues’ estimates in 10 of 35 predictions made (eFigure 8 in the Supplement). General surgery junior and senior residents did not differ significantly in their estimates (eFigure 9 in the Supplement). When comparing senior residents between the 2 specialties, no significant difference was found in their estimates (eFigure 10 in the Supplement).
This is the first study, to our knowledge, to address the accuracy of medical or surgical trainee assessment of operative risk in surgically complex patients compared with a national risk-adjusted model. Although the risk estimates for straightforward operations in otherwise healthy patients are available, patients with complex disease processes and high-risk premorbid states are conceptually more difficult to manage. Providing these patients and their families with a clear, individualized, accurate estimate of their surgical risk that accounts for preoperative cormorbid characteristics is necessary for them to make shared, informed treatment decisions.
Overall, internal medicine and general surgery residents had similar overestimates compared with the risk calculator and broad variability in their estimates. A previous study9 found that attending surgeons and residents have similar estimate accuracy, which may be a reflection of mirroring experiences from attending surgeons. From the internal medical standpoint, an overestimate of surgical risk could result in a decision not to request a surgical consultation and to opt for a palliative or nonoperative approach instead.
Of interest, no participants in this study underestimated risk of any of the scenarios. It is uncertain whether there is a general trend toward physician overestimation in practice, although a previous study10 suggests that internists overestimate risk in nonsurgical patients. The wide CIs in prediction demonstrate the difficulty in incorporating multiple preprocedural patient characterisitcs with the goal of individualizing risk prediction and counseling. This finding supports the importance of risk-adjusted models.
Surgery residents had significantly more confidence in their estimates even though the estimates were similarly erroneous between general surgery and internal medicine residents. This study could not elucidate a rationale behind more self-reported confidence among general surgery residents, but this finding may reflect more familiarity with the procedures compared with general medicine residents. Of interest, this study demonstrates that a general surgery resident’s familiarity with the proposed operations did not translate into more accurate risk estimates. Internal medicine residents reported being more likely to consult a risk-adjusted model, such as the NSQIP calculator, to estimate a patients’ operative risks and were more likely to discuss code status with their patients. These findings likely result from the different training paradigm in dealing with complex decision making and could represent opportunities for improvement in the curricula of both programs. For example, in response to these findings, we developed a curriculum that consisted of a 2-hour didactic session to assess current practice habits regarding risk prediction and patient counseling. In an active learning environment, participants use available risk-adjusted models to develop individualized risk profiles for patient scenarios and practice discussing the implications with patients. In addition, as a potential quality improvement initiative, we envision patients receiving an individualized risk profile report as part of their initial consultation with the surgeon and internist (who likely performs the preoperative risk stratification).
Despite the likely familiarity with these complex general surgical scenarios, general surgery residents were not superior in their estimation accuracy. General surgery residents were more comfortable not offering surgical intervention to those patients they deemed to have high risk of complications. This finding is concerning in the context of broad variability in the residents’ risk estimates and implies that an individual resident’s assessment and subsequent patient interaction could alter the decision to pursue an operative approach.
The decision ultimately lies with the attending surgeon, but a previous study9 found that general surgery residents and attending physicians are similarly inaccurate in their estimates. Of interest, there is no correlation between the perceived risk and the decision to offer surgery. For example, although participants predicted upward of 60% mortality in patients with cirrhosis requiring repair of incarcerated hernia, most offered surgery. This study does not elucidate the potential disconnect that we observed between perception of risk and decision to operate. Whether there exists a dose response curve of tolerance for risk for a given procedure is unknown and will be the subject of future study.
Junior and senior residents in both specialties consistently overestimated postoperative complications. Internal medicine junior residents were closer to the NSQIP predicted outcomes in 10 of 35 predictions. It is possible that senior resident clinical experience may have resulted in anectodal recall of a patient scenario in which outcomes were adverse, whereas internal medicine junior residents (interns) did not necessarily exhibit this bias. Regardless, the estimates were higher than the NSQIP risk-adjusted outcomes. This finding indicates the need for objective models that can provide a patient-individualized risk profile.
There are strengths and limitations to this study. Strengths include using real-life scenarios and a validated, risk-adjusted model representative of participating hospitals across the country. Evidence suggests that fidelity to real-life scenarios affect performance in simulated assessments.11 Furthermore, this study was the first, to our knowledge, to address the perception of risk and the subsequent decision to offer surgery and to incorporate a multidisciplinary approach to preoperative counseling.
Comparison of predictions with the NSQIP estimates may have limitations. For example, the NSQIP database includes operative cases only. In theory, this limitation could skew the results of the calculator toward underestimation of the true risk because patients selected for operation may be deemed to have acceptable risk vs high risk of complications. The calculator attempts to adjust for this possibility by permitting a surgeon adjustment option for when the patient has cofactors not captured by the risk calculator (such as anticoagulation status or immunosuppression other than use of steroids). This study did not use this adjustment because every case was designed to include only factors captured by the calculator. This decision not to use the surgeon adjustment may be an additional weakness of the study.
Another limitation of the study is the ability of the clinicians to physically evaluate the patient. There may be a role for the clinician’s physical examination and the surgeon instinct in judgment of fitness for surgery. There is no evidence to suggest that clinician intuition is superior to risk-adjusted models; nevertheless, the online scenario design did not permit this assessment. Even accounting for these potential shortcomings, it is evident that the wide CIs and variation in estimates suggest that clinicians are largely anectodal when estimating risk. Although the NSQIP calculator may not encompass all potential clinical scenarios, it is arguably the only available risk-adjusted model generalizable to a national patient population at present.
This study demonstrated that resident general surgeons and internal medicine residents, despite their differences, had similar assessments of the surgical risks in surgically complex patients. The study underscores the importance of availability and refinement of risk-adjusted models, such as the NSQIP risk calculator, to provide a consistent, individualized, and evidence-based assessment of surgical risk to patients. Nationally, there is an ongoing discussion in the literature regarding the accuracy of the risk calculator.12- 15 Despite the aforementioned limitations and the potential for underestimating risk, it is, to date, the most comprehensive, nationally representative model available. The estimates that it provides can offer an important starting point for discussion when surgically complex patients and their families are faced with the decision to undergo an operation. Additional studies are needed to evaluate general risk estimation practices across the United States and to validate the accuracy of currently available risk-adjusted models.
Corresponding Author: Kevin Pei, MD, Department of Surgery, Yale University School of Medicine, 330 Cedar St, BB310, New Haven, CT 06519 (firstname.lastname@example.org).
Accepted for Publication: July 2, 2017.
Published Online: October 11, 2017. doi:10.1001/jamasurg.2017.3936
Author Contributions: Drs Pei and Healy had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Healy, Pei.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Healy, Pei.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Healy.
Study supervision: Davis, Pei.
Conflict of Interest Disclosures: None reported.
Meeting Presentation: This study was presented at the annual meeting of the Association of VA Surgeons; May 7, 2017; Houston, Texas.