The complication rate for the Surgeon Scorecard (x-axis) was a composite measure of in-hospital mortality or related hospital readmission within 30 days of surgery, adjusted for patient age, comorbidity, and surgical approach (ie, open vs robotic). The models for the Michigan Urological Surgery Improvement Collaborative (y-axis) are adjusted for patient age, comorbidity, and surgical approach (ie, open vs robotic) and account for the clustering of outcomes within surgeons. A, Estimated blood loss. B, Surgical margin status for patients with organ-confined disease (pathologic stage = T2). Margin data are reported for 34 of 35 surgeons in the collaborative. C, Pelvic complications, defined by the Michigan Urological Surgery Improvement Collaborative as the occurrence of intraoperative rectal injury or the requirement for urinary catheter or pelvic drain for greater than 16 or 2 days, respectively. Data for this measure have been collected since April 1, 2014. D, Thirty-day rate of readmission or mortality.
Customize your JAMA Network experience by selecting one or more topics from the list below.
Auffenberg GB, Ghani KR, Ye Z, et al. Comparing Publicly Reported Surgical Outcomes With Quality Measures From a Statewide Improvement Collaborative. JAMA Surg. 2016;151(7):680–682. doi:10.1001/jamasurg.2016.0077
The recent release of a Surgeon Scorecard has accelerated debate around the merits of publicly reporting surgical outcomes.1 Based on Medicare claims from 2009 through 2013, this scorecard provides the public with surgeon-specific complication rates for 8 elective procedures performed by nearly 17 000 surgeons. While the intent of this effort—greater transparency leading to better outcomes—is laudable, many contend that the scorecard is misleading because it provides data for a single outcome measure that may not correlate well with other quality metrics.
We used data from the statewide clinical registry maintained by the Michigan Urological Surgery Improvement Collaborative (MUSIC) to evaluate this concern for one of the scorecard procedures—radical prostatectomy (RP) for prostate cancer. We specifically examined whether surgeon-specific complication rates reported in the scorecard correlate with several perioperative quality measures endorsed by MUSIC urologists and patient advocates.2
Established with support from the Blue Cross Blue Shield of Michigan, MUSIC is a consortium of 42 urology practices in Michigan that account for nearly 85% of urologists in the state. The collaborative maintains a prospective clinical registry, which includes detailed and validated intraoperative and postoperative clinical data obtained via medical record review by trained abstractors for all patients undergoing RP in participating practices.3
For this analysis, we identified every urologist in Michigan with both a risk-adjusted complication rate (ie, 30-day related readmission or in-hospital mortality) for RP in the Surgeon Scorecard released by ProPublica and outcome data for 10 or more RPs in the MUSIC registry.
We then fit multivariable models to estimate risk-standardized, surgeon-specific performance on several metrics tracked in MUSIC to assess technical quality and recovery after RP.2 These include blood loss, surgical margin status, pelvic complications, and 30-day readmissions and mortality. Using linear regression, we then examined the correlation between each MUSIC metric and the scorecard outcome. Statistical testing was performed at the 5% significance level using electronic software (StataCorp). Each practice obtained institutional review board approval of not-regulated status for collaborative participation.
Among the 48 surgeons from Michigan with complication rates reported in the scorecard, 40 participate in MUSIC, and 35 had data in the registry for at least 10 prostatectomies with more than 30 days of follow-up (Table). For this group (n = 35), case volumes in the scorecard and MUSIC registry ranged from 20 to 190 and 15 to 334, respectively.
Across 4 separate surgeon-specific outcomes, including a combined metric of 30-day readmissions and mortality analogous to that from the publicly reported data, we noted only weak, nonsignificant correlations (R2 range, 0.004-0.04) between measures from the MUSIC registry and complication rates from the scorecard (Figure).
For urologists in Michigan, we found no significant correlation between complication rates reported in the Surgeon Scorecard and perioperative quality measures from a statewide improvement collaborative. This finding supports a prevalent concern that the limited data available in the scorecard provide an incomplete, if not inaccurate, assessment of surgeon performance.4
However, there are other potential explanations for our findings. First, because dates for included surgical procedures were not entirely congruent between the data sources, the absence of correlations could reflect changes in surgical performance over time. Second, we only examined data for urologists in Michigan; stronger correlations may exist for surgeons from other regions or for other procedures.
Nonetheless, our findings highlight limitations with emerging public reporting initiatives. Quality-improvement collaboratives represent an established alternative that provide surgeons with comparative performance feedback on meaningful clinical and patient-reported outcomes while linking these measures with specific improvement initiatives.5 Urologists in Michigan have used this model to achieve population-level improvements in several aspects of prostate cancer care3 while avoiding potential inadvertent consequences of public reporting.6
Corresponding Author: David C. Miller, MD, MPH, Department of Urology, University of Michigan, 2800 Plymouth Rd, NCRC Bldg 16, Room 108E, Ann Arbor, MI 48109 (firstname.lastname@example.org).
Correction: This article was corrected on May 25, 2016, to fix an unclear description in the figure legend of the methods used by the Surgeon Scorecard.
Published Online: March 16, 2016. doi:10.1001/jamasurg.2016.0077.
Author Contributions: Drs Miller and Auffenberg had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Auffenberg, Ghani, Miller.
Acquisition, analysis, or interpretation of data: Auffenberg, Ye, Dhir, Gao, Stork, Miller.
Drafting of the manuscript: Auffenberg, Ghani, Miller.
Critical revision of the manuscript for important intellectual content: Ghani, Ye, Dhir, Gao, Stork, Miller.
Statistical analysis: Auffenberg, Ye, Gao.
Obtained funding: Ghani, Miller.
Administrative, technical, or material support: Ghani, Stork, Miller.
Study supervision: Miller.
Conflict of Interest Disclosures: Dr Auffenberg received grant support from the National Cancer Institute. Dr Ghani received contract support from the Blue Cross Blue Shield of Michigan (BCBSM) for serving as the codirector of the Michigan Urological Surgery Improvement Collaborative (MUSIC) and for serving as a consultant for Boston Scientific Corporation and Lumenis. Dr Miller received grant support from the National Cancer Institute and contract support from BCBSM for serving as the director of MUSIC and the Michigan Value Collaborative. No other disclosures were reported.
Funding/Support: Support for MUSIC is provided by BCBSM as part of the BCBSM Value Partnerships program. This work was also funded by grant T32-CA180984 from the National Cancer Institute.
Role of the Funder/Sponsor: Neither BCBSM nor the National Cancer Institute had a role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Additional Contributions: James E. Montie, MD (Department of Urology, University of Michigan, Ann Arbor), Chandy Ellimoottil, MD, MS (Department of Urology, University of Michigan, Ann Arbor), Edward Norton, PhD (Department of Health Management and Policy, University of Michigan, Ann Arbor), Scott Hawken, MS (Department of Urology, University of Michigan, Ann Arbor), and Susan Linsell, BS, MHSA (Department of Urology, University of Michigan, Ann Arbor), provided valuable advice on an early version of the manuscript. Drs Montie, Ellimoottil, and Norton and Mr Hawken and Ms Linsell were not compensated for their contributions. The authors acknowledge the significant contribution of the clinical champions, urologists, administrators, and data abstractors in each participating MUSIC practice (details around specific participating urologists and practices can be found at http://www.musicurology.com), as well as members of the MUSIC Coordinating Center at the University of Michigan. In addition, we acknowledge the support provided by David Share, MD, MPH (BCBSM), Tom Leyden, MBA (BCBSM), Rozanne Darland, BSBA (BCBSM), Karlie Witbrodt, MPH, MSW (BCBSM), and the Value Partnerships program at BCBSM. Dr Share, Mr Leyden, and Mss Darland and Witbrodt did not receive compensation for their contributions.