Improving Selection for Sentinel Lymph Node Biopsy Among Patients With Melanoma

This prognostic study/decision analytical model investigated the use of methodologies that improve expected prognostic accuracy and thus cost-effectiveness in selecting patients with melanoma for sentinel lymph node biopsy.


Introduction
The substantial cost of health care delivery constitutes a substantial burden on the economies of industrialized nations.Ample evidence points to the magnitude of waste when patients unnecessarily undergo costly medical procedures. 1 While medical societies develop guidelines to standardize approaches to medical care, these guidelines are mostly consensus based rather than evidence based and are inconsistently applied.They often fail to consider individual patient characteristics and differences among participating institutions.Strict reliance on established guidelines may therefore overlook patients who may benefit from a procedure while selecting some who are unlikely to benefit.Systematically refining guidelines to reflect relevant contextual factors concerning individual patients and institutions could substantially improve the chances of selecting the most appropriate patients.
7][8][9] Identification of lymph node metastases was revolutionized by the sentinel lymph node biopsy (SLNB) technique.2][13][14] While the procedure is generally well tolerated, it can be associated with complications, such as seroma, infection, and lymphedema.SLNB is typically recommended to patients based on eligibility criteria, including a primary tumor greater than 1.0-mm thick or thinner melanomas with high-risk features, such as ulceration or increased mitotic rate.The procedure identifies approximately 15% to 20% of patients as having node-positive melanoma. 5These patients derive most of the procedure's benefit (beyond the knowledge of node-negative status) given that patients with SLNB-negative melanoma are often treated in the same manner as if they had not undergone SLNB.This suggests that more accurate prediction of SLNB positivity could reduce the number of procedures performed and increase the number of patients with SLNB-positive melanoma identified, thus improving the cost-effectiveness of the selection procedure.Various nomograms have been developed to reflect risk of SLNB positivity. 15,16However, whether more accurate probabilistic algorithms could improve costeffectiveness in selecting patients to undergo SLNB and the extent of this improvement have not been investigated previously, to our knowledge.
We developed several novel statistical procedures, termed patient-centered methodology (PCM), that generate individually tailored probabilities assignable to specific clinical outcomes (eg, by stratifying patient cohorts into subgroups of differing risk for the outcome of interest and subsequently combining the effect of known prognostic factors within each separate subgroup). 17M has been shown to enable more accurate prediction of (1) survival outcomes in melanoma and breast cancer 17 and (2) the performance of mitotic rate in predicting prognoses in melanoma. 18This

Outcome Measures
We defined effectiveness as the appropriateness of patients selected to undergo SLNB, a staging procedure aiming to identify patients with node-positive melanoma.Thus, effectiveness was increased when whichever biopsies were performed detected more patients with node-positive melanoma.We defined cost as medical resources (including but not restricted to monetary expenditures) used in performing SLNBs.Thus, cost was reduced when fewer procedures were performed to identify patients with node-positive melanoma.We defined cost-effectiveness in terms of 2 outcomes from adopting minimum cutoff probabilities used to select patients to undergo SLNBs, assessed by comparing these outcome measures: actual or expected number of positive SLNBs achieved and required number of procedures performed to achieve that many positive SLNBs.

Statistical Analysis
PCM 17,18  Probabilistic algorithms generated by 3 different methodologies were applied to the 5989patient Australian cohort: the PCM-generated algorithm, an algorithm derived from conventional multiple logistic regression analysis, and an algorithm combining useful features of both methodologies (eMethods in Supplement 1).Selection of each eligible patient to undergo SLNB was simulated when that patient's SLNB-positive probability equaled or exceeded a range of minimally acceptable cutoff probabilities.The resulting cost-effectiveness of basing patient selection on different cutoffs was calibrated in terms of 2 outcome measures: actual or expected number of positive SLNBs achieved and the required number of procedures performed to achieve that number.
Cost-effectiveness was separately assessed using SLNB-positivity probabilities generated by each of the 3 algorithms produced by each methodology, respectively.The goodness of fit of each algorithm's probabilities was also assessed by comparing the observed number of positive SLNBs with its expected number, and corresponding R 2 values were calculated.Data were analyzed using SPSS statistical software version 27 (IBM) from October 2000 to April 2021.
Initially, we assessed the utility of 12 prognostic factors in estimating individual patient SLNBpositivity probabilities in the Australian cohort using conventional multiple logistic regression.In stepwise analysis of these factors, all except sex were independently associated with SLNB outcome; χ 2 ranged from 3.88 (P = .05)for mitotic rate to 65.14 (P < .001)for melanoma subtype (Table 2), with a combined AUROC of 0.753 (Figure 1A).
Next, we used PCM to produce comparable probabilities of SLNB positivity.We stratified the Australian cohort into 3 prognostic subgroups based on tumor thickness and patient age, 2 covariates associated with the greatest changes in SLNB outcome (Table 2); 79 of 893 patients in the We then fitted by separate logistic regression analysis an algorithm that best predicted SLNB positivity from the same 12 prognostic factors to each patient subgroup.Distinct subsets of factors were identified as independent covariates in each subgroup (eTables 3-5 in Supplement 1).
Prognostic probabilities from each subgroup were merged into a composite index, which achieved an AUROC of 0.803 (Figure 1A).To verify PCM's increased accuracy, matched pairs of absolute value error differences in predicting SLNB outcomes were assessed, with significant differences favoring smaller PCM errors in the Wilcoxon matched-pair signed-ranks test and the binomial sign test (P < .001).PCM produced a more accurate prognosis 69.4% more frequently than when using conventional logistic regression analysis.
Separately, we examined US patients who had undergone SLNB.There were statistically significant differences in the composition of the 2 cohorts (Table 1), with the Australian cohort exhibiting a higher proportion of patients with thicker melanomas and a higher SLNB-positivity rate.
In addition, 3 prognostic factors (regression, melanoma subtype, and TIL grade) were coded in  different ways, precluding simple merging of data from cohorts and complicating the application of data from 1 cohort to the other.We assessed the prognostic utility of the same 12 prognostic factors with conventional logistic regression analysis, which achieved an AUROC of 0.777 (Figure 1B).As in our analysis of the Australian cohort, we used PCM to generate low-, intermediate-, and high-risk subgroups, with 38 of 597 patients (6.4%), 87 of 458 patients (19.0%), and 112 of 287 patients (39.0%) experiencing a positive SLNB outcome, respectively, with significant differences by group as shown in the χ 2 test (P < .001)eTable 2 in Supplement 1).After merging of probabilities for 3 subgroups, the composite index achieved an AUROC of 0.826 (Figure 1B).The improved accuracy of PCM in predicting SLNB outcomes compared with conventional logistic regression analysis was supported by a matched-pairs analysis of absolute errors (P < .001for signed rank and binomial tests).PCM produced a more accurate prognosis 100.9% more frequently than conventional analysis.
We then assessed the cost-effectiveness of selecting patients to undergo SLNB by virtue of equaling or exceeding a minimum cutoff probability.Separate SLNB-positivity probabilities were generated by 3 separate algorithms.To the 3640 patients in the Australian cohort, we added 2349 patients (eTable 1 in Supplement 1) who were eligible for but did not undergo SLNBs.Probabilities were produced for all 5989 Australian patients from the 2 corresponding algorithms derived from the 3640 Australian patients and 1342 US patients who underwent SLNB.An adequate sample of eligible patients who did not undergo the procedure could not be identified in the US cohort, precluding a similar analysis for that cohort.
Using the PCM-generated algorithm, adopting an 8.7% minimum cutoff probability of SLNB positivity resulted in the same number of simulated SLNBs as actually performed (3640 SLNBs) but with 1066 expected positive outcomes, for a positivity rate of 29.3%.This constituted an improvement of 287 SLNBs compared with 779 actual positive SLNBs (36.8% improvement).In contrast, adopting a 23.7% minimum cutoff resulted in performing 1825 simulated SLNBs, with the same expected number of positive outcomes (779 SLNBs), for a positivity rate of 42.7%, and requiring 1815 fewer simulated procedures (49.9%).Each minimum cutoff probability that produced curve 1 of Figure 2A between the previously described 2 reference probabilities illustrated costeffective dominance (ie, larger numbers of expected positive SLNB outcomes and smaller numbers of procedures required to achieve these positive outcomes compared with the actual experience).A, A comparison is presented of the cost-effectiveness of various sentinel lymph node biopsy (SLNB)-positive probability algorithms used to guide simulated selection of patients to undergo SLNB, all applied to the Melanoma Institute Australia (MIA) cohort.Curve 1 applies the patient-centered methodology-generated algorithm from the Australian cohort, curve 2 applies the algorithm developed using conventional methodology from the Australian cohort, and curve 3 applies the SLNB outcome algorithm from the US cohort to the same patients in the Australian cohort.B, Concordance is presented between patient-centered methodology-generated expected counts of positive SLNB outcomes (based on patient-centered methodologygenerated probabilities) and the actual recorded counts of positive SLNB outcomes.
In addition, we assessed results when applying 4 PCM-generated minimum-cutoff SLNBpositivity reference probabilities that may be applied in practice (range, 5%-20%).Melanoma centers typically recommend SLNB to an individual patient when the probability of a positive outcome ranges between 5% and 10%.Adopting 10%, 15%, or 20% PCM-generated minimum cutoffs improved both outcome measures simultaneously (given that they were within the costeffective dominance range of 8.7% and 23.7%) (Table 3).For example, the PCM cutoff of 10% produced 1046 expected positive SLNBs of 3417 simulated SLNBs performed (36.0%), constituting an improvement of 267 more than the 779 positive SLNBs actually achieved (34.3% improvement) and a reduction of 223 unneeded SLNBs from the 3640 procedures actually performed (6.1% reduction).In contrast, adopting the 5% minimum cutoff probability increased the number of expected positive SLNB outcomes, resulting in a SLNB-positivity rate of 1123 of 4636 SLNBs (24.2%), for an increase of 344 positive SLNBs (44.2%); however, it required 996 additional expected procedures to be performed(27.4%)(Table 3).All possible PCM-generated minimum cutoff probabilities and associated pairs of SLNB outcomes are provided in eTable 6 in Supplement 2, ranked by ascending positivity probabilities.
We next assessed the cost-effectiveness of judicious patient selection when cutoff probabilities were generated from conventional multiple logistic regression analysis of the same 12 prognostic factors in the MIA cohort.Use of these probabilities in simulation also resulted in improved costeffectiveness compared with actual experience, although with smaller differences than found when using probabilities generated by PCM (Figure 2A).The range of cost-effective dominant minimum cutoff probabilities decreased to between 11.3% and 19.8%.Thus, 5%, 10%, and 20% reference cutoffs were outside the dominance range (Figure 2A and Table 3) and therefore did not generate unambiguously superior selections.For example, the conventional cutoff of 10% resulted in an Minimum cutoff probabilities between 0.087 and 0.237 were cost-effective dominant vs the actual experience (ie, there were nonnegative numbers for increased positivity and reduced SLNBs).
b The conventional algorithm was a logistic regressiongenerated algorithm.Probabilities are listed in eTable 7 in Supplement 2. Minimum cutoff probabilities between 0.113 and 0.198 were costeffective dominant vs the actual experience (ie, there were nonnegative numbers for increased positivity and reduced SLNBs).
c The mixed algorithm was a PCM and conventional logistic regression-generated algorithm.Probabilities are listed in eTable 8 in Supplement 2. Minimum cutoff probabilities between 0.105 and 0.174 were cost-effective dominant vs the actual experience (ie, there were nonnegative numbers for increased positivity and reduced SLNBs).
expected positivity rate of 1044 of 3968 SLNBs performed (26.3%), constituting an improvement of 265 positive SLNBs vs the actual experience (34.0%improvement) but an increase of 328 unneeded SLNBs (9.0%).All possible minimum cutoff probabilities and associated pairs of SLNB outcomes generated by this analysis are provided in eTable 7 in Supplement 2. Despite significant differences in composition of the 2 cohorts, we sought to investigate whether a probability-estimating algorithm derived from 1 cohort could improve the costeffectiveness of selecting patients in the other cohort.A prognostic algorithm was produced from the US cohort by combining PCM and conventional logistic regression analysis.Application of this algorithm to the 5989-patient MIA cohort also improved the cost-effectiveness of selecting patients to undergo SLNB compared with actual experience (Figure 2A), albeit with a smaller improvement than with other algorithms.Specific outcomes achieved using the same reference minimum cutoff probabilities are shown in Table 3

Discussion
Melanoma is expected to represent the second most common malignant tumor in the US by 2040. 19 estimated mean cost of more than $25 000 20 per SLNB in the US translates to substantial current and anticipated future costs.Given that approximately 80% of patients who currently undergo SLNB do not have lymph node metastases, selecting appropriate patients for SLNB in a cost-effective manner may become increasingly important and worthwhile.
This hybrid prognostic study/decision analytical model found improved discrimination in predicting SLNB positivity.These results extend prior such attempts.Investigators at the Memorial Sloan Kettering Cancer Center developed a nomogram using 5 prognostic factors that achieved an AUROC of 0.694. 15More recently, Lo and colleagues at Melanoma Institute Australia 16 developed a nomogram using 6 prognostic factors and reported an AUROC of 0.739.The incorporation of gene expression-profiling assays may be associated with further improvement in prognostic discrimination, but to date no detailed comparative studies have been reported. 21,22The greater prognostic discrimination achieved in our study (AUROCs of 0.826 and 0.803 in US and Australian cohorts, respectively) was realized by using PCM to estimate a separate, individually and institutionally tailored probability of SLNB positivity based on 12 established and routinely available clinical and histopathological prognostic factors.Several features of PCM enabled its increased capability in our models, 17,18 including stratifying the population into more homogeneous risk subgroups, developing indices for each prognostic factor reflecting the shape of its impact on the outcome measure of interest, and special handling of missing observations.
The improved prognostic capability of PCM was exploited to enable improvements in the costeffectiveness of selecting patients to undergo SLNB.Patients were selected by comparing their estimated SLNB-positive probability with a minimum cutoff probability.The PCM-based selection procedure produced an expanded range of cost-effectiveness dominance compared with that obtained using conventional logistic regression analysis.Both methodologies improved costeffectiveness compared with the actual experience in the Australian cohort.Intriguingly, using a 5% cutoff probability (which is used by many melanoma centers) resulted in outcomes that were outside the PCM cost-effective dominance range, whereas using a 10% value produced outcomes within the range.This suggests that appropriate minimum cutoff probabilities recommended by various guidelines may usefully be revisited. 23r results suggest substantial room for improvement in the cost-effectiveness of the process by which patients with melanoma are currently selected to undergo SLNB.To achieve this improvement, each health system may need to choose its preferred minimum cutoff probability, with the goal of reducing the total number of patients undergoing SLNB (by up to 49.9% in the Australian cohort) or increasing the number of patients with node-positive melanoma detected (by up to 36.8% in this cohort) or some balance between these goals.Minimum cutoff probabilities and resulting pairs of SLNB outcomes indicated the consequences of applying the chosen cutoff probability using each algorithm, while outcomes found using cutoff probabilities and cost-effectiveness curves indicated whether that probability was within each algorithm's cost-effective dominance range.
An important possibility is that with a sufficiently large sample of patients required to generate stable probabilistic estimates, each health system may develop an algorithm tailored to the composition of its own patient population and reflecting the distinct ways various prognostic factors are recorded.Any such locally tailored (or externally obtained) algorithm may be incorporated into an interactive electronic (eg, web-based) tool whose inputs are specific prognostic factors and whose outputs are calculated SLNB-positive probabilities for local individual patients.We recommend that the National Comprehensive Cancer Network and other national melanoma guideline committees consider these issues in discussing minimum cutoff probabilities.At the individual patient level, this may have important implications for the decision-making process.For example, the conversation with a male patient aged 34 years with a 0.7-mm acral melanoma without TILs (harboring >10% risk) would be markedly different than that with a female patient aged 87 years with a 1.8-mm desmoplastic melanoma on the upper arm (harboring <5% risk).

Limitations
This study has several limitations, including compositional differences between cohorts and the difficulty in applying an algorithm from 1 institution to another when relevant prognostic factors were not similarly coded.This analysis constitutes a retrospective simulation demonstrating the potentially improved cost-effectiveness achievable by selecting various cutoff probabilities.
However, a prospective clinical trial would be required to confirm improvements realized by adopting a chosen minimum cutoff probability to select patients for SLNB using PCM or any other appropriate prognostic methodology.

Conclusions
Beyond SLNB for melanoma, results of this hybrid prognostic study/decision analytical model study may have important implications for how guidelines are developed for selecting patients to undergo other useful medical procedures.These results suggest the consequences that choosing different minimum cutoff probabilities may have for the cost-effectiveness of the selection process and the unambiguous advantages that may be gained by selecting patients within the cost-effective dominance range when one is identified.Importantly, this approach reflects a conceptual advance beyond a strictly categorical approach (ie, whether a patient is eligible to undergo a given procedure) to a comparative approach (ie, which patients are more eligible to undergo that procedure) in contexts in which differing degrees of eligibility are sensible and ascertainable.

Figure 1 .
Figure 1.Discriminating Among Sentinel Lymph Node Biopsy Outcomes Using Various Prognostic Methodologies

Figure 2 .
Figure 2. Cost-effectiveness of Applying Various Algorithms and Concordance Between Expected and Actual Outcomes Improving Selection for Sentinel Lymph Node Biopsy Among Patients With Melanoma JAMA Network Open.2023;6(4):e236356.doi:10.1001/jamanetworkopen.2023.6356(Reprinted) April 19, 2023 2/13 Downloaded From: https://jamanetwork.com/ on 09/24/2023 study investigated PCM predictions of individualized SLNB positivity compared with other methods and the resulting cost-effectiveness in selection of patients to undergo SLNB.cohort).Criteria used to define eligibility and recommend SLNB included melanomas greater than 1.0 mm thick or 1.0 mm or less in thickness with ulceration or a mitotic rate of 1 per mm 2 or greater.Covariates Information available on 12 prognostic factors was used for subsequent analyses.These factors were patient age, patient sex, tumor site, tumor thickness, ulceration, mitotic rate, presence of microsatellites, presence of regression, tumor-infiltrating lymphocyte (TIL) grade, presence of lymphatic invasion, tumor type, and Clark level.

Table 1 .
Characteristics of Australian and US Cohorts (continued) -risk group (8.9%) were SLNB positive.This increased to 239 of 1355 patients in the intermediaterisk group (17.6%) and 461 of 1392 patients in the high-risk group (33.1%), with significant differences in SLNB-positivity rates by group as shown by χ 2 test (P < .001)(eTable 2 in Supplement 1). low

Table 2 .
Conventional Analysis of Expected Impact of Factors on SLNB Positivity a a Conventional analysis consisted of stepwise multiple logistic regression with backward elimination.Analysis was among 3640 patients in the Melanoma Institute Australia cohort.b χ 2 values indicate deviations separating each regression-estimated numeric coefficient characterizing, respectively, the association of each independent variable with the single dependent variable from its uniformly null-hypothesisstipulated value of zero.Each tabled P value corresponds to its corresponding null-hypothesized zero-value statistical test.

Table 3 .
Simulated Impact on Cost-effectiveness of Using Different Methodologies to Select Patients to Undergo SLNB vs Actual Experience a Probabilities are listed in eTable 6 in Supplement 2.

eTable 4 .
Stepwise Multiple Logistic Regression of Impact of Factors on Sentinel Lymph Node Biopsy Positivity in 1355 Patients in Australian Intermediate-Risk Subset eTable 5. Stepwise Multiple Logistic Regression of Impact of Factors on Sentinel Lymph Node Biopsy Positivity in 1392 Patients in Australian High-Risk Subset Patient-Centered Methodology-Generated Minimum Cutoff Probabilities and Associated Pairs of Patient Selection Outcomes in Australian Cohort eTable 7. Minimum Cutoff Probabilities Generated by Conventional Analysis and Associated Pairs of Patient Selection Outcomes in Australian Cohort eTable 8. Minimum Cutoff Probabilities Generated by Mixed Analysis and Associated Pairs of Patient Selection Outcomes in Australian Cohort