eAppendix. Sample report.
Customize your JAMA Network experience by selecting one or more topics from the list below.
Albertini JG, Wang P, Fahim C, et al. Evaluation of a Peer-to-Peer Data Transparency Intervention for Mohs Micrographic Surgery Overuse. JAMA Dermatol. 2019;155(8):906–913. doi:10.1001/jamadermatol.2019.1259
Can the use of benchmarked peer-to-peer reports reduce overuse in Mohs micrographic surgery (MMS)?
In this nonrandomized controlled intervention study including 2329 US surgeons, sharing individual reports of MMS practice patterns with physicians resulted in an immediate and sustained reduction in mean stages per case among outlier physicians and an associated cost savings compared with physicians who did not receive these reports.
A nonpunitive, confidential audit-and-feedback intervention may represent an opportunity to reduce overuse of MMS among physicians identified as outliers by their clinical specialty society.
Mohs micrographic surgery (MMS) is a skin cancer treatment that uses staged excisions based on margin status. Wide surgeon-level variation exists in the mean number of staged resections used to treat a tumor, resulting in a cost disparity and question of appropriateness.
To evaluate the effectiveness of a behavioral intervention aimed at reducing extreme overuse in MMS, as defined by the specialty society, by confidentially sharing stages-per-case performance data with individual surgeons benchmarked to their peers nationally.
Design, Setting, and Participants
This nonrandomized controlled intervention study included 2329 US surgeons who performed MMS procedures from January 1, 2016, to March 31, 2018. Physicians were identified using a 100% capture of Medicare Part B claims. The intervention group included physicians affiliated with the American College of Mohs Surgery, and the control group included physicians not affiliated with the American College of Mohs Surgery.
Individualized performance reports were delivered to all outlier surgeons, defined by the specialty society as those with mean stages per case 2 SDs above the mean, and inlier surgeons in the intervention group.
Main Outcomes and Measures
The primary outcome was surgeon-level change in mean stages per case between the prenotification (January 2016 to January 2017) and postnotification (March 2017 to March 2018) periods. A multivariable linear regression model was used to evaluate the association of notification with this surgeon-level outcome. The surgeon-level metric of mean stages per case was not risk adjusted. The mean Medicare cost savings associated with changes in practice patterns were calculated.
Of the 2329 included surgeons, 1643 (70.5%) were male and 2120 (91.0%) practiced in metropolitan areas. In the intervention group (n = 1045), 53 surgeons (5.1%) were outliers; in the control group (n = 1284), 87 surgeons (6.8%) were outliers. Among the outliers in the intervention group, 44 (83%) demonstrated a reduction in mean stages per case compared with 60 outliers in the control group (69%; difference, 14%; 95% CI of difference, −1 to 27; P = .07). There was a mean stages-per-case reduction of 12.6% among outliers in the intervention group compared with 9.0% among outliers in the control group, and outliers in the intervention group had an adjusted postintervention differential decrease of 0.14 stages per case (95% CI, −0.19 to −0.09; P = .002). The total administrative cost of the intervention program was $150 000, and the estimated reduction in Medicare spending was $11.1 million.
Conclusions and Relevance
Sharing personalized practice pattern data with physicians benchmarked to their peers can reduce overuse of MMS among outlier physicians.
Skin cancer is the most common malignancy in the United States, accounting for more than 5.4 million cases annually at an estimated cost of more than $8.1 billion to the US health care system.1,2 Mohs micrographic surgery (MMS) is a surgical approach that treats skin cancer using a potential series of staged excisions determined by intraoperative histologic examination. Mohs micrographic surgery is an effective and often curative treatment for cutaneous basal and squamous cell carcinoma.3,4 In addition, the procedure is highly efficient, since the surgeon serves the simultaneous roles of oncologic surgeon, pathologist, and generally also reconstructive surgeon. Mohs micrographic surgery use has increased over the last decade owing to the procedure’s superior outcomes, its expanding application to melanoma, an emphasis of MMS in training programs, and increased rates of skin cancer.5-12
We have previously described wide variation among surgeons performing MMS in the mean number of stages used to resect a skin cancer.11 Potential overuse may stem from insufficient technical expertise or the current fee-for-service payment model, which compensates surgeons based on the number of stages performed per tumor, creating a perverse incentive for a surgeon to use an excessive number of staged resections to remove a lesion. Overuse of stages per case burdens patients with unnecessary surgical resections and taxes the health care system with avoidable costs.11 For these reasons, the American College of Mohs Surgery (ACMS), the largest specialty society for MMS, considers a surgeon’s annual mean stages per case as an appropriateness metric for Mohs surgery of the head and neck. To address this disparity in appropriateness and cost, we designed an audit-and-feedback intervention aimed at reducing extreme overuse in MMS by confidentially sharing stages-per-case performance data with individual surgeons benchmarked to their peers nationally for similar head and neck lesions. We examined the association of the intervention with a surgeon’s practice pattern and the overall cost to Medicare.
We conducted a nonrandomized controlled intervention study of surgeons in the United States who billed MMS procedures to Medicare. The intervention group consisted of all ACMS members, and they received an individualized notification that reported their annual mean stages per case benchmarked to national peer performance. The control group consisted of non-ACMS members who did not receive a notification. The prenotification performance period was January 1, 2016, through January 31, 2017. The intervention group received an individualized notification in February 2017. The postnotification period was March 1, 2017, through March 31, 2018. We excluded surgeons who performed 10 or fewer MMS procedures in either the prenotification or postnotification period, as required by our user agreement with Medicare designed to protect patient identity. This study received waived authorization of institutional review board approval by the Johns Hopkins Medical Institutions Institutional Review Board and informed consent was not needed because this study was based on administrative claims data only and did not involve active recruitment of study participants.
The Improving Wisely collaborative is a Robert Wood Johnson Foundation–funded initiative that uses national data to identify overuse patterns for various diagnostic, medical, and surgical procedures. In the Improving Wisely model, an overuse metric is developed by expert consensus, and Medicare data are queried to identify physician outliers and inliers.13 Confidential, benchmarked, personalized audit-and-feedback data reports are shared with physicians in an effort to change physician behavior and reduce overuse and cost waste.
As part of the Improving Wisely collaborative, ACMS created a Physician Engagement Council of highly respected national experts and clinical leaders, including the association’s executive leadership as elected by the ACMS membership, and representation from large, small, urban, and rural practice settings. Through this workgroup, the ACMS developed and endorsed a consensus threshold of what constitutes an outlier physician practice pattern of overuse. This consensus definition of an outlier was defined as a physician’s annual mean stages per case for MMS for skin cancers of the head, neck, genitalia, hands, and feet being greater than 2 SDs above the national mean.11 This metric was developed specifically for this study, taking into consideration our inability to control for surgeon case mix.
In the month prior to the notification intervention, national leaders in the specialty introduced the Improving Wisely collaborative and objectives in a printed and electronic newsletter. In addition, surgeons in the intervention cohort also received an email notification detailing the upcoming confidential, personalized, audit-and-feedback report 2 weeks in advance of the mailing. All physicians in the intervention cohort were mailed their individualized performance report using the most recent addresses listed with their professional association. Accompanying the individual performance data, a cover letter signed by national peer leaders in the specialty described the benchmarking approach and the goals of the program.
Each report listed the individual surgeon’s mean number of stages per case, the national mean of stages per case, and the consensus metric definition and included a cover letter explaining the Improving Wisely program and the intervention (eAppendix in the Supplement). A graphical representation of an individual surgeon’s performance was depicted using an arrow pointing to the surgeon’s position on the national distribution. Surgeons were classified into 3 groups that were differentiated by color; green represented surgeons considered inliers (ie, performance fell within the acceptable range of clinical variation), yellow represented surgeons whose mean stages per case fell within 1.0 to 2.0 SDs above the national mean, and red represented surgeons considered outliers (ie, performance was more than 2.0 SDs above the national mean). Surgeons who fell in the lattermost category also had the word “outlier” appear on their report next to their mean stages per case along with their percentile rank.
Communications from the ACMS president to the membership at the time of notification described the Improving Wisely reports and invited surgeons to contact the leadership with any questions or concerns. Confidentiality, collegiality, respect, and a spirit of quality improvement were emphasized in all communications and announcements about the study. A follow-up letter was sent to all outlier members offering confidential mentoring and educational resources, including a slide deck on surgical and laboratory techniques and select readings to improve one’s proficiency.
Mean stages per case for head and neck procedures were calculated from the Physician Supplier Part B claims file accessed via the US Centers for Medicare & Medicaid Services Data Center.14 This file includes all fee-for-service claims submitted on a CMS-1500 claims form and includes the claims from physicians and free-standing facilities. We identified the first stage by Current Procedural Terminology (CPT) code 17311 and additional stages by CPT code 17312. The mean stages per case were calculated by dividing the sum of the first and additional stages by the number of first stages (ie, [total CPT codes 17311 + 17312]/CPT code 17311).
We identified surgeon characteristics, including sex, practice location, and region, from the Medicare Data on Provider Practice and Specialty.15 We obtained surgeon year of medical school graduation from the Physician Compare National Downloadable File.16 Surgeon information was linked to claims data using National Provider Identifier numbers.
The primary outcome was surgeon-level change in mean stages per case in the post-notification period compared with the prenotification period. Cost was defined as the Medicare-allowed amount in US dollars paid to surgeons directly for MMS (professional fee only). The cost savings for each outlier was calculated by multiplying their number of cases performed during the postnotification period by their change in mean stages per case and by their mean reimbursed amount for each additional stage (CPT code 17312) during the postnotification period. We calculated the mean cost savings per case for each group by dividing the total cost savings by the total case volume in the postnotification period.
We compared physician characteristics between intervention and control groups using 2-sample t tests for continuous variables and χ2 tests for categorical variables. We used paired t tests to compare surgeons’ prenotification and postnotification mean stages per case within each arm of the study. We used independent 2-sample t tests to compare surgeons’ change in mean stages per case between the intervention and control groups stratified by outlier status. A power calculation was not performed because our study included the entire population of eligible MMS surgeons who billed to Medicare in the specified time period.
We used a multivariable linear regression model to evaluate factors associated with surgeon-level change in mean stages per case.16,17 We modeled surgeons’ postnotification minus prenotification change in mean stages per case as a continuous outcome, with the independent variables being surgeon sex, years since medical school graduation, practice region and location, case volume in the prenotification period, outlier status, notification status, and an interaction term for outlier × notification status. All P values were from 2-sided tests, and results were deemed statistically significant at a P value less than .05. All statistical analyses were performed using SAS Enterprise Guide version 7.1 (SAS Institute).
The number of eligible MMS surgeons in the United States was 2329 (Table 1). There were 140 outliers (53 in the intervention group and 87 in the control group) and 2189 inliers (992 in the intervention group and 1197 in the control group). The outliers in the intervention group paralleled the outliers in the control group in regards to mean stages per case in the prenotification period, male predominance, and metropolitan location of practice (Table 1). Compared with outliers in the control group, intervention group outliers had fewer years in practice (median time, 16 vs 22 years), a higher procedural volume (median prenotification volume, 423 vs 102 cases), and were more likely to practice in the northeast region of the United States (38% [20 of 53] vs 15% [13 of 87]).
Following the intervention, outliers in the intervention group demonstrated a decrease in mean stages per case in the first quarter of 2017, which was sustained through the first quarter of 2018 (Figure 1). Outliers in the control group demonstrated a smaller decrease in mean stages per case beginning in the second quarter of 2017, which continued for the remainder of the study period. Following the intervention, the overall mean stages per case decreased from 2.55 to 2.31 among outliers in the intervention group and decreased from 2.56 to 2.46 among outliers in the control group. Outlier surgeons in the intervention group exhibited a mean reduction of 0.26 stages per case, whereas outliers in the control group exhibited a mean reduction of 0.11 stages per case, corresponding to a differential reduction of 0.14 stages per case (95% CI, −0.23 to −0.05; P = .002) (Table 2). Of the 53 intervention group outliers notified, 44 (83%) demonstrated a reduction in mean stages per case in the postnotification period (median [range] reduction, 0.21 [0.05-1.22] stages per case). In contrast, 60 of 87 control group outliers (69%) demonstrated a decrease in mean stages per case in the postnotification period (median [range] reduction, 0.20 [0.002-0.94] stages per case), a difference of 14% in rate of response (95% CI of difference, −1 to 27; P = .07) (Figure 2). Among the 53 intervention group outliers, 9 surgeons (17%) demonstrated a median (range) increase of 0.06 (0.02-0.16) stages per case compared with 27 control group outliers (31%) with a median (range) increase of 0.08 (0.001-0.45) stages per case. Inlier surgeons in the intervention group exhibited a mean reduction of 0.03 stages per case, whereas inliers in the control group exhibited a mean reduction of 0.01 stages per case, corresponding to a differential reduction of 0.01 stages per case (95% CI, −0.03 to −0.003; P = .01) (Table 2).
In the multivariable linear regression model that adjusted for baseline surgeon characteristics (Table 3), the intervention was associated with an additional 0.14 reduction in mean stages per case (95% CI, −0.19 to −0.09; P < .001) among outliers and an additional 0.02 reduction in mean stages per case (95% CI, −0.03 to −0.001; P = .03) among inliers. The intervention was associated with greater change among outliers compared with inliers (coefficient for interaction, −0.13; 95% CI, −0.18 to −0.07). Holding other factors equal, surgeons practicing for less than 10 years since graduation decreased a mean of 0.03 stages per case (95% CI, −0.05 to −0.01) more than those practicing for 30 years or more since graduation. Surgeon sex, practice region and location, and case volume were not significantly associated with a change in mean stages per case.
Excluding research salaries, approximately $150 000 ($144 per surgeon) was used to cover the costs of the intervention. Using the mean stages per case in the prenotification period as a reference, surgeons’ stage reduction in the postnotification period was associated with a total cost savings of $11 143 882 to Medicare. This included a savings of $2 458 672 by the outliers in the intervention group and $788 262 by the outliers in the control group as well as a savings of $5 114 177 by the intervention group inliers and $2 782 771 by the control group inliers. The mean cost savings per case was $95 for intervention group outliers vs $46 for control group outliers and $11 for intervention group inliers vs $10 for control group inliers.
Engaging physicians to promote behavior change around best practices has been a significant barrier to reducing waste in health care.13,18-22 The emerging field of implementation science seeks to address this problem and has categorized quality improvement interventions by their approach. These categories include audit and feedback, use of local opinion leaders, reminders, and educational sessions.23-27 Of these approaches, interventions that use opinion leaders and those that use audit and feedback have been observed to have the greatest association with physician behavior change. In a 2012 meta-analysis of 70 studies using the audit-and-feedback approach,23 the Cochrane Effective Practice and Organization of Care group described a 1% to 4% change in desired physician behavioral outcomes. Similarly, a 2011 Effective Practice and Organization of Care review of 18 cluster randomized clinical trials involving 296 hospitals27 found that using opinion leaders in an intervention results in 9% to 14% change in desired behavior.
We combined 2 established approaches of using audit and feedback and opinion leaders, which resulted in an immediate and sustained reduction in physician behavior around low-value care in MMS. Among outliers in the intervention group, 83% demonstrated a reduction in mean stages per case compared with 69% of outliers in the control group. There was a mean reduction in stages per case of 12.6% among outliers in the intervention group compared with 9.0% among outliers in the control group. Adjusted for physician baseline characteristics, the intervention was associated with an additional reduction of 0.14 stages per case (95% CI, −0.19 to −0.09; P < .001) among outliers. We posit that the improvement we observed in this study may be attributed to 4 elements characteristic of the intervention: (1) endorsement of an overuse metric by a clinical specialty society, (2) awareness and education by the specialty society around overuse, (3) the use of an achievable target, and (4) a confidential, nonpunitive audit-and-feedback comparison process.
The potential latent crossover effect observed among control group outliers may be attributed to 2 factors. First, the educational efforts and increased awareness efforts by the specialty society at the time of the intervention may have influenced surgeons in the control group. We believe that having peers from the specialty association prepare members in the intervention group to receive their notification reports was critical to surgeons being receptive to them. Second, clinicians in the intervention group may have had an impact on control group participants through personal conversations or a perception of an increased culture of accountability fostered by the initiative. Further, we observed a small yet statistically significant decrease in stages per case among inliers in the intervention group, suggesting that while outliers were more likely to demonstrate a greater reduction in mean stages per case, inliers may have also modified their behavior. Notably, even a slight decrease in stages per case in the inlier group could result in a significant cost savings, given that outlier volume only comprises a small percentage of all MMS procedures.
Physician demand for benchmarking data has been described to be strong. A 2011 study of primary physicians28 revealed that 95% of physicians surveyed believe there is excessive variation in care, and 76% are interested to learn how their individual practice patterns compare with their peers. We posit these views are shared by other specialist groups and hypothesize that additional iterations of Improving Wisely report dissemination could elicit even greater reductions in overuse.29
This study and its approach through the Improving Wisely model may have implications for broader quality improvement and waste reduction efforts in health care.13 We have identified critical factors that are likely needed for subsequent replication of this design. First, as a prerequisite, the intervention requires a well-respected peer physician leadership group to endorse a metric and the broader initiative a priori.30,31 In the present study, the physician engagement council and ACMS endorsed the overuse measure and broader program. Second, a clear achievable benchmark that is supported by peer leaders in the field is required, with a consensus threshold of what constitutes outlier practice patterns. In this study, experts created a consensus threshold of 2 SDs above the national mean. Third, the addition of educational and training resources as intervention components may be valuable to help outliers seeking tools to improve. Finally, the manner in which reports are messaged and delivered to the physician is important.32 Our methods were consistent with audit-and-feedback delivery best practices.33 Specifically, we ensured the reports were easily accessible, short in length and easy to interpret (ie, reduced cognitive load), and provided a combination of text and visual feedback.33 The reports were nonpunitive and confidential, provided clear targets for the desired behavior, were endorsed by respected leaders and governing bodies, provided repeated messaging and communication to targeted surgeons, and emphasized a behavior that was in the control of individual surgeons. This concept of autonomy is critical, with data suggesting that behavior change effects are stronger if an individual believes they are acting autonomously rather than simply responding to external influences.34-37
This study has several limitations. First, while claims data are powerful for benchmarking and not subject to reporting bias, they do not provide detailed clinical data, such as patient or tumor characteristics. While our proposed metric adopted an appropriateness range based on expert consensus regarding a surgeons’ typical case mix, it is possible that some outliers’ high mean stages per case might be attributed to a high percentage of large and complex cases and/or high-risk patients in their practice. Despite this, the intervention group still showed a greater reduction in mean stages per case. Additional data sources would expand the analysis beyond the Medicare population. Detailed registries, such as the ACMS Clinical Data Registry (MohsAIQ) or the American Academy of Dermatology DataDerm Registry, may facilitate risk-adjusted comparisons and may enable individual physicians to examine their mean number of stages across all payers and compare themselves to benchmarks more easily in the future. Additional research will be needed to determine whether any unintended negative consequences resulted from the reduction in mean stages per case, such as tumor recurrence. Second, our study intervened on a single metric, which addressed only 1 aspect of appropriateness in MMS. However, additional metrics (eg, appropriate use of MMS for trunk and extremities, complications, and recurrences) can and should be explored.38,39 Further, we emphasize that the metric used to identify outliers was developed for the purpose of this study, in consideration of the claims data limitations. Future studies should revisit these benchmarks and provide adjustments for those clinicians with extensive case mix. Third, to facilitate intervention delivery, a nonrandomized design was used. While we adjusted for some physician characteristics, differences between surgeons in the intervention and control groups, including history of fellowship training and clinical practice setting, could lead to residual confounding. Future quality improvement initiatives should aim to target a representative sample of surgeons performing MMS. Fourth, it is unknown which educational resources were used by intervention group outliers and which resources may have contributed to the improvement observed. The sustainability and durable impact of this intervention should be explored in the long term.
We observed an immediate and sustained reduction in MMS procedures using an intervention that integrated the use of experts, a specialty association partnership, and educational resources along with individualized, confidential data report cards that provided a benchmark for peer comparison. The relatively low cost of this intervention relative to its resultant cost savings suggests that application to other areas of medicine could yield larger savings to the health care system. In an era where reducing low-value care is increasingly paramount given the escalating costs of health care, this intervention has the promise to provide a high return on investment.
Accepted for Publication: April 14, 2019.
Corresponding Author: John G. Albertini, MD, The Skin Surgery Center, 1450 Professional Park Dr, Ste 150, Winston Salem, NC 27103 (email@example.com).
Published Online: May 5, 2019. doi:10.1001/jamadermatol.2019.1259
Author Contributions: Dr Makary had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Albertini, Hutfless, Stasko, Vidimos, Coldiron, Bennett, Marks, Overton, Bruhn, Xu, Krishnan, Makary.
Acquisition, analysis, or interpretation of data: Albertini, Wang, Fahim, Hutfless, Stasko, Vidimos, Leshin, Billingsley, Park, Bruhn, Xu, Krishnan.
Drafting of the manuscript: Albertini, Wang, Fahim, Hutfless, Park, Makary.
Critical revision of the manuscript for important intellectual content: Albertini, Wang, Fahim, Hutfless, Stasko, Vidimos, Leshin, Billingsley, Coldiron, Bennett, Marks, Overton, Bruhn, Xu, Krishnan.
Statistical analysis: Wang, Hutfless, Park, Xu, Krishnan.
Obtained funding: Albertini, Makary.
Administrative, technical, or material support: Fahim, Vidimos, Bruhn, Makary.
Study supervision: Albertini, Fahim, Bennett, Xu, Makary.
Conflict of Interest Disclosures: Dr Albertini has received grants and nonfinancial support from the American College of Mohs Surgery as well as personal fees from Novascan for serving on a scientific advisory board and serves on the board of directors and is a shareholder of QualDerm Partners. Dr Leshin has received personal fees from Novascan for serving on a scientific advisory board and is a shareholder of QualDerm Partners. Dr Overton is supported by grant 5T32CA126607-10 from the National Institutes of Health. Dr Makary has received grants from the Robert Wood Johnson Foundation. No other disclosures were reported.
Funding/Support: This study was funded by a grant from the Robert Wood Johnson Foundation.
Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Meeting Presentation: This paper was presented at the 2019 Annual Meeting of the American College of Mohs Surgery; May 5, 2019; Baltimore, MD.
Create a personal account or sign in to: