Overall crude revision rates were 10.6% for both the observed and risk-adjusted revisions (vertical dashed lines). Error bars indicate 95% CIs.
Customize your JAMA Network experience by selecting one or more topics from the list below.
Rudmik L, Xu Y, Alt JA, et al. Evaluating Surgeon-Specific Performance for Endoscopic Sinus Surgery. JAMA Otolaryngol Head Neck Surg. 2017;143(9):891–898. doi:10.1001/jamaoto.2017.0752
When using a risk-adjusted, 5-year surgery revision rate as a surrogate marker for performance, are there differences in surgeon-specific performance for endoscopic sinus surgery?
After evaluating 43 surgeons within the province of Alberta, Canada, there were differences in surgeon-specific performance for endoscopic sinus surgery. Three variables demonstrated significant associations with 5-year endoscopic sinus surgery revision rate: presence of nasal polyps, more annual systemic corticosteroid courses, and concurrent septoplasty.
Evaluating surgeon-specific performance for endoscopic sinus surgery may provide information to assist in quality improvement. Given the findings from this study, the surgeon-specific, risk-adjusted, 5-year endoscopic sinus surgery revision rate may represent a quality metric to assess surgical performance during management of chronic rhinosinusitis.
Several identified factors have raised questions concerning the quality of care for endoscopic sinus surgery (ESS), including the presence of large geographic variation in the rates and extent of surgery, poorly defined indications, and lack of ESS-specific quality metrics. Combined with the risk of major complications, ESS represents a high-value target for quality improvement.
To evaluate differences in surgeon-specific performance for ESS using a risk-adjusted, 5-year ESS revision rate as a quality metric.
Design, Setting, and Participants
This retrospective study used a population-based administrative database to study adults (≥18 years of age) with chronic rhinosinusitis (CRS) who underwent primary ESS in Alberta, Canada, between March 1, 2007, and March 1, 2010. The study period ended in 2015 to provide 5 years of follow-up.
Endoscopic sinus surgery for CRS.
Main Outcomes and Measures
Primary outcomes were the 5-year observed and risk-adjusted ESS revision rate. Logistic regression was used to develop a risk adjustment model for the primary outcome.
A total of 43 individual surgeons performed primary ESS in 2168 patients with CRS. Within 5 years after the primary ESS procedure, 239 patients underwent revision ESS, and the mean crude 5-year ESS revision rate was 10.6% (range, 2.4%-28.6%). After applying the risk adjustment model and 95% CI to each surgeon, 7 surgeons (16%) had lower-than-expected performance and 2 surgeons (5%) had higher-than-expected performance. Three variables had significant associations with surgeon-specific, 5-year ESS revision rates: presence of nasal polyps (odds ratio [OR], 2.07; 95% CI, 1.59-2.70), more annual systemic corticosteroid courses (OR, 1.33; 95% CI, 1.19-1.48), and concurrent septoplasty (OR, 0.70; 95% CI, 0.55-0.89).
Conclusions and Relevance
Evaluating surgeon-specific performance for ESS may provide information to assist in quality improvement. Although most surgeons had comparable risk-adjusted, 5-year ESS revision rates, 16% of surgeons had lower-than-expected performance, indicating a potential to improve quality of care. Future studies are needed to evaluate more surgeon-specific variables and validate a risk adjustment model to provide appropriate feedback for quality improvement.
Health care systems are focused on improving the quality and value of health care delivery.1-3 Because of increasing technology costs, constrained surgical resources, and large practice variation, there is significant interest in measuring the performance of individual surgeons to drive quality improvement.4-6 Information obtained from measuring and appropriately risk adjusting surgeon performance would be useful to various stakeholders. This information might give surgeons real-time individual feedback, assist patients to make informed decisions about their care, and allow third-party payers to link remuneration with quality outcomes. However, the measurement of surgeon performance for endoscopic sinus surgery (ESS) lags behind other medical specialties.7,8 A recent study by Smith et al6 evaluated surgeon-specific outcomes for ESS among 3 academic institutions, and the results provided the first insights into how outcomes can vary among surgeons. When conducted appropriately, evaluating surgeon-specific performance may represent an important strategy to improve the overall quality of the health care system.9-13
It is well accepted that ESS can provide significant short- and long-term benefit to patients with refractory chronic rhinosinusitis (CRS).14-16 With more than 250 000 operations performed each year in the United States,17 this volume of ESS rivals inguinal hernia repair and breast lumpectomy and exceeds the rates of pacemaker placement and colorectal resections.18 Recently, several identified factors have raised questions concerning the quality of care for ESS, including the presence of large geographic variation in the rates and extent of surgery,19-22 poorly defined indications,23 and lack of ESS-specific quality metrics.24 These factors, combined with the risk of major complications,25,26 make ESS a high-value target for quality improvement, and measuring individual surgeon performance may assist in further quality improvement.
The objective of this study was to evaluate surgeon-specific performance for ESS by using a risk-adjusted, 5-year ESS revision rate as a quality metric. Evaluation of differences in adjusted surgeon-specific ESS revision rates may provide novel insight into how quality of care may vary during management of refractory CRS.
The Data Integration, Measurement, and Reporting (DIMR) administrative database of Alberta Health Service was used to obtain data for this study. The DIMR database collects physician claims on all government-funded health care encounters in Alberta, Canada. There is no private provision of health care for CRS in Alberta; therefore, all medical care for CRS is captured in the DIMR database. Inclusion criteria for this study were adults (≥18 years of age) with CRS (defined by validated case definition27) in Alberta who underwent primary ESS between March 1, 2007, and March 1, 2010 (3-year time horizon). Because more than 95% of ESS revision cases occur within 5 years after the primary ESS procedure,28 this study evaluated a 5-year follow-up time. An ESS revision was defined as an additional ESS procedure within 5 years after the primary ESS date. Patients were excluded from the final analysis if they had an ESS claim before 2007. An ESS case was defined as a minimum physician claim of a maxillary antrostomy (Alberta procedure code 34.1A) and ethmoidectomy (Alberta procedure code 34.54A). The use of physician claims to identify cases of ESS was validated in a prior study.19 Conjoint Health Research Ethics Board approval from the University of Calgary was obtained, no informed consent was required, and all data were deidentified.
All patients included in this analysis were cross-linked to the providing surgeon’s anonymous practice identification number along with the Pharmaceutical Information Network and the Discharge Abstract Database to collect drug- and comorbidity-related variables. Because physician claims were used to identify ESS, all cases of ESS revision were included regardless of whether the additional operation was performed by the primary ESS surgeon.
The outcome variable of interest was the 5-year observed and risk-adjusted ESS revision rates. Logistic regression was used to develop a risk adjustment model for the 5-year ESS revision rate. Multiple potential confounding variables were included in the risk adjustment model (Box). Race/ethnicity is only reported in aggregate per geographic region in the DIMR database; therefore, the regression analysis could not include this variable. On the basis of the 2011 Canadian Census, most of the Alberta population is white (75%), with the next visible minorities being Aboriginal (6%), South Asian (4%), Chinese (3%), and black (2.7%); therefore, the association of potential geographic variation with race/ethnicity was determined to be minimal. To assess the association of the extent of surgery with outcomes, we assumed that the addition of a frontal sinusotomy claim (Alberta procedure code 34.32B or 34.32C) indicated that the surgeon performed a complete ESS as opposed to a limited ESS when only maxillary antrostomy (Alberta procedure code 34.1A) and ethmoidectomy (Alberta procedure code 34.54A) claims were submitted. To maintain surgeon anonymity in this study, the only surgeon factor included in the model was annual ESS case volume. Other identifiable variables, such as age, year of training, fellowship training status, and location, were not included to ensure surgeon anonymity.
Geographic region (urban or rural)
Chronic obstructive pulmonary disease
Congestive heart failure
Topical intranasal steroid treatment (No. of bottles per year)
Oral antibiotic treatment (No. of courses per year)
Oral corticosteroid treatment (No. of courses per year)
Underwent a concurrent septoplasty with ESS
No. of ENT visits before ESS
Annual surgeon primary ESS case volume
Frontal sinusotomy (marker for complete ESS)
Abbreviations: ENT, ear, nose, and throat; ESS, endoscopic sinus surgery.
The model determined the probability of an ESS revision for each patient based on the combination of clinical risk variables present in the patient. The expected ESS revision rate for each surgeon was then calculated using the mean probabilities from the model. The calculated observed and expected ESS revision rate for each surgeon was used to generate surgeon-specific observed-expected revision ratios. To obtain risk-adjusted revision rates, we then multiplied the surgeon-specific observed-expected revision ratios by the overall revision rate for the 43 surgeons. The 95% CIs of the observed and adjusted revision rates were derived by applying the binomial method. The surgeon-specific adjusted ESS revision rate was considered to be higher or lower than the expected rate when the 95% CI did not cross the crude observed revision rate.
The association between the adjusted variables and surgeon-specific, 5-year ESS revision rate is measured by odds ratio (OR) and 95% CI. The performance of the logistic regression model was assessed using the C statistic, and a value of greater than 0.7 was considered to be a good model.29 Because of the multiple variables included in the logistic regression model (n = 24 variables), the Bonferroni correction was applied to reduce the risk of a type I error (ie, incorrectly rejecting the null hypothesis). For this study, a 2-sided P ≤ .002 was considered statistically significant.
A total of 43 surgeons performed primary ESS in 2168 patients for CRS. The mean number of ESS cases per surgeon within the 3-year study period was 50 (range, 16-167). Within 5 years after the primary ESS, 239 patients underwent revision ESS, and the mean crude 5-year ESS revision rate was 10.6% (range, 2.4%-28.6%) (Figure).
The associations between the adjusted variables and surgeon-specific, 5-year ESS revision rate as measured by OR and 95% CI are reported in Table 1. With use of P ≤ .002 to account for multiple comparisons, the following 3 variables demonstrated significant associations with surgeon-specific, 5-year ESS revision rates: presence of nasal polyps (OR, 2.07; 95% CI, 1.59-2.70), more annual systemic corticosteroid courses (OR, 1.33; 95% CI, 1.19-1.48), and concurrent septoplasty (OR, 0.70; 95% CI, 0.55-0.89).
Risk-adjusted, 5-year ESS revision rates per surgeon are shown in the Figure. After applying the 95% CI to each surgeon’s adjusted revision rate, 2 surgeons (5%) had lower-than-expected revision rates (ie, performed better than expected) and 7 surgeons (16%) had higher-than-expected revision rates (ie, performed lower than expected). The mean annual volume of primary ESS cases for the top 2 performing surgeons was 26 cases compared with a mean annual volume of 10 primary ESS cases (range, 5-19 cases) for the 7 surgeons with lower-than-expected performance. Although low subgroup sample size precludes the ability to perform meaningful statistical analysis, the rates for each of the 3 patient characteristics that were significantly associated with 5-year ESS revision rate for the top- and low-performing surgeons are reported in Table 2.
This study evaluated the differences in surgeon-specific, 5-year ESS revision rates for CRS. The results demonstrate that the crude 5-year ESS revision rate for CRS is 10.6%, which is similar to previously reported ESS revision rates.30,31 When evaluating the performance of individual surgeons, the results indicate that after risk adjusting, most surgeons had comparable outcomes, with 7 surgeons (16%) having higher-than-expected ESS revision rates (ie, lower than expected performance). The finding that nasal polyps and more systemic corticosteroid courses negatively affect ESS revision is expected because the presence of nasal polyps has been associated with higher rates of ESS revisions in a prior study28 and surgeons tend to prescribe systemic corticosteroids for nasal polyps. Furthermore, the finding that concurrent septoplasty positively affected the rate of revision ESS (ie, reduced rate) was similar to the results from a recent US-based study.6 Using larger surgeon sample sizes, future studies need to determine whether there are other variables that need to be accounted for in the risk-adjustment model before surgeon performance can be evaluated using revision rate as a quality metric.
In theory, with expanding research into the pathophysiology and clinical outcomes for CRS combined with improved surgical technology, the quality of care for this chronic disease should have increased during the past 3 decades. However, despite substantial advances in the management of CRS, persistent gaps still exist in the quality of care for CRS, such as large practice variation,19,20,32,33 delayed access to ESS,34,35 and postoperative complications that require hospital care.25 As we continue to search for ways to improve how we deliver care for CRS, it is important to begin evaluating the role that surgeons play in this quality-improvement puzzle. Although quality improvement is a highly complex process that involves multiple stakeholders, evaluating the performance of surgeons and providing feedback in real time represent a promising strategy.4 Although other medical and surgical specialties have started registries focused on performance measurement,10,36-38 research focused on evaluating the performance of surgeons during management of CRS has been limited.
A variety of metrics can reasonably be examined when looking at surgeon-specific outcomes for management of CRS. This study examined the ESS revision rate, but other potential metrics include degree of quality-of-life improvement, work productivity, need for subsequent medications and physician visits, cost of surgical treatment, or prevention of CRS progression. In 2016, a US-based study by Smith et al6 evaluated surgeon-specific outcomes for CRS across 3 academic institutions. The results indicated that there were unadjusted differences in patient outcomes across the 3 institutions, and the differences appeared to be driven by variations in the degree of baseline CRS-specific quality-of-life impairment, oral corticosteroid dependence, and concurrent septoplasty. Objective measures of CRS (ie, computed tomography and endoscopy) failed to explain the differences in patient outcomes among institutions and were not included in the risk-adjustment model, a finding that suggests that objective disease severity was not driving the differences in surgeon performance. The effects of other patient characteristics, such as depression, prior ESS, and acetylsalicylic acid sensitivity, were not assessed because the prevalence was similar among all 3 institutions. The primary limitation of the study by Smith et al6 was that it only included surgeons from 3 tertiary-level practices and did not include community-based practices. This model reduces the generalizability of the findings to a broader population-level perspective of care for CRS.
Although the study by Smith et al6 represents the first insights into how outcomes can vary across academic institutions, the current study evaluated 43 different surgeons across a Canadian province and includes tertiary-level and community-based rhinology practices. Overall, the results support the notion that it is feasible to compare surgeon-specific performance for ESS after appropriate risk adjustment has been applied. Although most of the surgeons had similar outcomes, 7 surgeons (16%) had lower-than-expected performance when using an adjusted 5-year ESS revision rate with the 95% CI as a quality metric. This finding indicates that there is variation in surgical outcomes, and delivery of ESS to patients with CRS can be improved.
Three variables were identified to affect surgeon performance outcomes and thus may be important to include in risk-adjustment models for ESS. First, the presence of nasal polyps was associated with higher rates of ESS revision, and this finding has been supported by several previous studies.28,30,39 Second, receipt of more systemic corticosteroid courses before ESS was associated with a higher risk of ESS revision. This finding may be attributable to patients with nasal polyps often being treated with courses of systemic corticosteroids before ESS and therefore having a higher risk of ESS revision. Another potential reason may indicate that patients treated with multiple courses of systemic corticosteroids had a delay in undergoing ESS and thus may have worse long-term outcomes.34 Third, a concurrent septoplasty was associated with a reduced risk of undergoing an ESS revision. Potential explanations for this finding may include patients breathing better after ESS, having improved access to topical medical therapies after ESS, or perhaps undergoing septoplasty that augmented the technical success of ESS by allowing better access to the sinuses. On the other hand, concurrent septoplasty may be a confounding variable for patients receiving technically more complete primary ESS and appropriate postoperative care because surgeons who are more comfortable performing ESS (ie, sinus surgeons performing a large volume of operations and/or fellowship-trained rhinologists) may have a lower threshold to perform a concurrent septoplasty. Although it is unknown why concurrent septoplasty positively affects surgeon performance by using the ESS revision rate, the study by Smith et al6 evaluated only the performance of fellowship-trained rhinologists, and the results also indicated that performing a concurrent septoplasty positively affected outcomes after ESS. Although it is important to prevent unnecessary procedures, failure to perform a concurrent septoplasty when a deviated septum is present may negatively affect outcomes after ESS.
Although most surgeons had similar adjusted ESS revision rates, this study found that 7 surgeons (16%) had higher-than-expected revision rates, suggesting there may be an opportunity to improve the care provided by these surgeons. However, this study does not provide an answer as to why these surgeons have higher rates of revision. Because of the need to maintain surgeon anonymity, this study was unable to evaluate more detailed surgeon variables, such as age, fellowship training status, practice location (ie, rural or urban), surgical technique, or quality of postoperative care.40 Despite the limitations of trying to elucidate reasons for the lower-than-expected performance from 7 surgeons, variables such as annual ESS case volume and fellowship training status may warrant future investigation using larger surgeon sample sizes. First, we were able to evaluate the annual case volume of ESS per surgeon without compromising surgeon anonymity. Although there was no observed or adjusted association between overall surgeon volume and 5-year ESS revision rate in the logistic regression analysis, an examination of the extremes revealed that the top-performing surgeons had higher annual case volumes (mean annual ESS case volume, 17) compared with surgeons with lower performance (mean annual ESS case volume, 8). Because of the known effect of surgeon volume on patient outcomes from other procedures,41,42 future studies should continue to evaluate the effect of ESS volumes using larger sample sizes. Second, the low-performing surgeons had a lower rate of concurrent septoplasty during primary ESS (Table 2). Although the significance of concurrent septoplasty needs further investigation, current evidence suggests that concurrent septoplasty is associated with reduced risk of ESS revision. Although there are several potential reasons to explain the differences in surgeon performance, this study provides insight into understanding that differences in surgeon-specific outcomes exist for ESS and there is a need to identify factors that lead to lower performance. Once factors that affect surgeon performance are identified, they can be used to improve the quality of care.
Despite the possibility to compare surgeons based on adjusted 5-year ESS revision rates, it is important to emphasize that the model used in this study may not represent the most accurate risk adjustment for surgeon comparison. Specifically, because of the limitations of using administrative data, there may be unaccounted variables that are outside the surgeon’s control that were not included in the risk-adjustment model. For example, the degree of baseline quality-of-life impairment has a significant effect on treatment choice43 and surgical outcomes44-46 and therefore may need to be used to adjust patient outcomes after ESS. The risk-adjustment model in this study failed to include baseline or follow-up quality-of-life data because the database did not collect these variables. Furthermore, additional factors that are often not collected in administrative data must be considered when evaluating surgeon performance for ESS, for example, patient adherence to postoperative medical therapy, access to appropriate surgical equipment (eg, image guidance), and measurement of whether surgeons assessed patient preferences during the decision-making process or ensured that patients had appropriate expectations for performing revision ESS. Because risk adjustment is critical to enable appropriate and transparent comparison of surgeon performance for CRS, future studies will need to evaluate the effect of other potential confounding variables in risk-adjustment models.
Another limitation of this study is the inability to include cases of ESS revision that occurred outside Alberta. However, because there is no private health care for CRS in other provinces of Canada and Alberta has 2 tertiary-level otolaryngology institutions at which complex revision ESS procedures are performed, it would be rare for patients from Alberta to seek care outside their province.
Last, the intended and unintended consequences that could arise from reporting an ESS quality metric such as 5-year ESS revision rate must be considered. Positive benefits would arise if surgeons with high ESS revision rates critically evaluate their practices and make quality improvements. Patients are also more likely to benefit when ESS, including revision, is performed for appropriate indications by well-trained surgeons. From a negative perspective, surgeons may start to cream-skim by inappropriately adjusting their practice to deny primary ESS to patients at higher risk of additional operations, such as those with nasal polyps or other severe CRS phenotypes. Surgeons could also game the quality metric by not offering revision surgery to their own patients who might otherwise benefit from surgery or by being more liberal in offering revision surgery to patients who underwent primary surgery performed by another surgeon. These and other issues will need to be addressed before implementation of comparing surgeon performance for ESS.
This study evaluated differences in surgeon-specific performance for ESS. Although most surgeons had comparable risk-adjusted, 5-year ESS revision rates, 16% of surgeons had lower-than-expected performance, indicating a potential to improve quality of care. The presence of nasal polyps and more prior systemic corticosteroid courses were associated with increased risk of undergoing ESS revision, whereas concurrent septoplasty was associated with a reduced risk of ESS revision. Future studies need to evaluate more surgeon-specific variables and validate a risk-adjustment model to provide appropriate feedback for quality improvement.
Corresponding Author: Luke Rudmik, MD, MSc, Division of Otolaryngology–Head and Neck Surgery, Department of Surgery, University of Calgary, 1820 Richmond Rd SW, Calgary, AB T2T 5C7, Canada (email@example.com).
Accepted for Publication: April 17, 2017.
Published Online: June 22, 2017. doi:10.1001/jamaoto.2017.0752
Author Contributions: Drs Rudmik and Xu had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Rudmik, Xu, Alt, DeConde, Quan.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Rudmik, Alt, DeConde, Smith, Schlosser.
Critical revision of the manuscript for important intellectual content: Rudmik, Xu, Alt, DeConde, Smith, Quan, Soler.
Statistical analysis: Rudmik, Xu, Quan.
Obtained funding: Rudmik.
Administrative, technical, or material support: Alt, DeConde, Smith, Soler.
Study supervision: Rudmik, Alt, DeConde, Smith, Schlosser, Quan.
Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr Rudmik reported serving on the scientific advisory board for BioInspire and 480 Biomedical. Dr Alt reported serving as a consultant for Medtronic, Spirox, and GlycoMira Therapeutics. Dr Deconde reported serving as a consultant for IntersectENT and Stryker Endoscopy. Dr Schlosser reported serving as a consultant for Olympus, Medtronic, and Arrinex and receiving grant support from Intersect and Entellus. Dr Soler reported serving as a consultant for Olympus Inc and 480 Biomedical Inc. No other disclosures were reported.
Funding/Support: This study was supported by an MSI Foundation Grant and a Petro-Canada Young Innovator in Community Health Sciences of Canada Award (Dr Rudmik).
Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and the decision to submit the manuscript for publication.
Additional Contributions: We thank the MSI Foundation and Petro-Canada Young Innovator in Community Health Sciences of Canada Award for the grants to support this project.