[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
Figure 1.
Observed and Predicted Numbers of Incident Melanomas by Deciles of Predicted Risk Over 20 Years of Follow-up Using Australian Rates From 2007 Through 2009
Observed and Predicted Numbers of Incident Melanomas by Deciles of Predicted Risk Over 20 Years of Follow-up Using Australian Rates From 2007 Through 2009

This graph compares the observed and predicted numbers of incident melanomas by deciles of predicted risk over 20 years of follow-up.

Figure 2.
Decision Curves Obtained From Plotting the Net Benefit at Different 20-Year Absolute Risk Thresholds
Decision Curves Obtained From Plotting the Net Benefit at Different 20-Year Absolute Risk Thresholds

This graph compares the decision curves from classifying individuals as high risk using the risk prediction model, classifying all individuals as high risk, and classifying all individuals as low risk (horizontal line at 0) over 20 years of follow-up. Net benefit at different 20-year absolute risk thresholds is calculated as {true-positive classifications – [% risk threshold/(100 − % risk threshold) × false-positive classifications]}/total number of participants (see the Results section).

Table.  
Relative Riska Estimates for Risk Factors in the Melanoma Risk Prediction Model in the Development and Independent Validation Studies
Relative Riska Estimates for Risk Factors in the Melanoma Risk Prediction Model in the Development and Independent Validation Studies
Supplement.

eFigure 1. Nevus density pictograms used in the Australian Melanoma Family Study

eTable 1. Australian melanoma incidence and competing mortality rates per 100,000 by age group and sex from 2007-2009

eTable 2. Distributions of the risk factors in the melanoma risk prediction model in the development and independent validation studies

eMethods 1. Area under the receiver operating curve (AUC) after reweighting the age and sex distribution of the case-control study controls to the general population

eTable 3. Area under the receiver operating curve weights (AUC) in the Western Australia Melanoma Study

eTable 4. Area under the receiver operating curve (AUC) weights in the Leeds Melanoma Case-Control Study

eTable 5. Reweighted area under the receiver operating curve (AUC) in the Western Australia Melanoma Study and Leeds Melanoma Case-Control Study

eMethods 2. Model recalibration using Swedish melanoma incidence and mortality rates to estimate 20 year absolute risk

eTable 6. Melanoma incidence and mortality without melanoma per 100,000 by age group for Swedish women 2009-2011

eTable 7. Melanoma incidence and mortality without melanoma per 100,000 by age group for Swedish women 1991-2011

eFigure 2. Observed and predicted numbers of incident melanomas by deciles of predicted risk over 20 years of follow up using Swedish rates from 2009-2011

eFigure 3. Observed and predicted numbers of incident melanomas by deciles of predicted risk over 20 years of follow up using Swedish rates from 1991-2011

eReferences

1.
Erdmann  F, Lortet-Tieulent  J, Schüz  J,  et al.  International trends in the incidence of malignant melanoma 1953-2008—are recent generations at higher or lower risk?  Int J Cancer. 2013;132(2):385-400.PubMedGoogle ScholarCrossref
2.
Stanton  WR, Janda  M, Baade  PD, Anderson  P.  Primary prevention of skin cancer: a review of sun protection in Australia and internationally.  Health Promot Int. 2004;19(3):369-378.PubMedGoogle ScholarCrossref
3.
Ahmed  H, Naik  G, Willoughby  H, Edwards  AG.  Communicating risk.  BMJ. 2012;344:e3996.PubMedGoogle ScholarCrossref
4.
Freedman  AN, Seminara  D, Gail  MH,  et al.  Cancer risk prediction models: a workshop on development, evaluation, and application.  J Natl Cancer Inst. 2005;97(10):715-723.PubMedGoogle ScholarCrossref
5.
Vuong  K, McGeechan  K, Armstrong  BK, Cust  AE.  Risk prediction models for incident primary cutaneous melanoma: a systematic review.  JAMA Dermatol. 2014;150(4):434-444.PubMedGoogle ScholarCrossref
6.
Olsen  CM, Neale  RE, Green  AC, Webb  PM, Whiteman  DC; QSkin Study; Epigene Study.  Independent validation of six melanoma risk prediction models.  J Invest Dermatol. 2015;135(5):1377-1384.PubMedGoogle ScholarCrossref
7.
Collins  GS, de Groot  JA, Dutton  S,  et al.  External validation of multivariable prediction models: a systematic review of methodological conduct and reporting.  BMC Med Res Methodol. 2014;14:40.PubMedGoogle ScholarCrossref
8.
Cust  AE, Schmid  H, Maskiell  JA,  et al.  Population-based, case-control-family design to investigate genetic and environmental influences on melanoma risk: Australian Melanoma Family Study.  Am J Epidemiol. 2009;170(12):1541-1554.PubMedGoogle ScholarCrossref
9.
English  DR, Armstrong  BK.  Identifying people at high risk of cutaneous malignant melanoma: results from a case-control study in Western Australia.  Br Med J (Clin Res Ed). 1988;296(6632):1285-1288.PubMedGoogle ScholarCrossref
10.
Newton-Bishop  JA, Chang  YM, Iles  MM,  et al.  Melanocytic nevi, nevus genes, and melanoma risk in a large case-control study in the United Kingdom.  Cancer Epidemiol Biomarkers Prev. 2010;19(8):2043-2054.PubMedGoogle ScholarCrossref
11.
Newton-Bishop  JA, Chang  YM, Elliott  F,  et al.  Relationship between sun exposure and melanoma risk for tumours in different body sites in a large case-control study in a temperate climate.  Eur J Cancer. 2011;47(5):732-741.PubMedGoogle ScholarCrossref
12.
Kvaskoff  M, Pandeya  N, Green  AC,  et al.  Site-specific determinants of cutaneous melanoma: a case-case comparison of patients with tumors arising on the head or trunk.  Cancer Epidemiol Biomarkers Prev. 2013;22(12):2222-2231.PubMedGoogle ScholarCrossref
13.
Olsen  CM, Green  AC, Neale  RE,  et al; QSkin Study.  Cohort profile: the QSkin Sun and Health Study.  Int J Epidemiol. 2012;41(4):929-929i.PubMedGoogle ScholarCrossref
14.
Roswall  N, Sandin  S, Adami  HO, Weiderpass  E.  Cohort profile: the Swedish Women’s Lifestyle and Health cohort [published online June 10, 2015].  Int J Epidemiol. doi:10.1093/ije/dyv089.PubMedGoogle Scholar
15.
Veierød  MB, Adami  HO, Lund  E, Armstrong  BK, Weiderpass  E.  Sun and solarium exposure and melanoma risk: effects of age, pigmentary characteristics, and nevi.  Cancer Epidemiol Biomarkers Prev. 2010;19(1):111-120.PubMedGoogle ScholarCrossref
16.
Gandini  S, Sera  F, Cattaruzza  MS,  et al.  Meta-analysis of risk factors for cutaneous melanoma: I. common and atypical naevi.  Eur J Cancer. 2005;41(1):28-44.PubMedGoogle ScholarCrossref
17.
Gandini  S, Sera  F, Cattaruzza  MS,  et al.  Meta-analysis of risk factors for cutaneous melanoma: II. sun exposure.  Eur J Cancer. 2005;41(1):45-60.PubMedGoogle ScholarCrossref
18.
Gandini  S, Sera  F, Cattaruzza  MS,  et al.  Meta-analysis of risk factors for cutaneous melanoma: III. family history, actinic damage and phenotypic factors.  Eur J Cancer. 2005;41(14):2040-2059.PubMedGoogle ScholarCrossref
19.
Cust  AE, Armstrong  BK, Goumas  C,  et al.  Sunbed use during adolescence and early adulthood is associated with increased risk of early-onset melanoma.  Int J Cancer. 2011;128(10):2425-2435.PubMedGoogle ScholarCrossref
20.
Chang  YM, Barrett  JH, Bishop  DT,  et al.  Sun exposure and melanoma risk at different latitudes: a pooled analysis of 5700 cases and 7216 controls.  Int J Epidemiol. 2009;38(3):814-830.PubMedGoogle ScholarCrossref
21.
Vergouwe  Y, Royston  P, Moons  KG, Altman  DG.  Development and validation of a prediction model with missing predictor data: a practical approach.  J Clin Epidemiol. 2010;63(2):205-214.PubMedGoogle ScholarCrossref
22.
Gail  MH, Brinton  LA, Byar  DP,  et al.  Projecting individualized probabilities of developing breast cancer for white females who are being examined annually.  J Natl Cancer Inst. 1989;81(24):1879-1886.PubMedGoogle ScholarCrossref
23.
Bruzzi  P, Green  SB, Byar  DP, Brinton  LA, Schairer  C.  Estimating the population attributable risk for multiple risk factors using case-control data.  Am J Epidemiol. 1985;122(5):904-914.PubMedGoogle Scholar
24.
Steyerberg  EW.  Clinical Prediction Models. New York, NY: Springer; 2010.
25.
Veierød  MB, Weiderpass  E, Thörn  M,  et al.  A prospective study of pigmentation, sun exposure, and risk of cutaneous malignant melanoma in women.  J Natl Cancer Inst. 2003;95(20):1530-1538.PubMedGoogle ScholarCrossref
26.
Steyerberg  EW, Vergouwe  Y.  Towards better clinical prediction models: seven steps for development and an ABCD for validation.  Eur Heart J. 2014;35(29):1925-1931.PubMedGoogle ScholarCrossref
27.
Vickers  AJ, Elkin  EB.  Decision curve analysis: a novel method for evaluating prediction models.  Med Decis Making. 2006;26(6):565-574.PubMedGoogle ScholarCrossref
28.
Royston  P, Moons  KG, Altman  DG, Vergouwe  Y.  Prognosis and prognostic research: developing a prognostic model.  BMJ. 2009;338:b604.PubMedGoogle ScholarCrossref
29.
Bender  R, Lange  S.  Adjusting for multiple testing—when and how?  J Clin Epidemiol. 2001;54(4):343-349.PubMedGoogle ScholarCrossref
30.
Collins  GS, Reitsma  JB, Altman  DG, Moons  KG.  Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement.  BMJ. 2015;350:g7594.PubMedGoogle ScholarCrossref
31.
Pepe  MS, Fan  J, Seymour  CW, Li  C, Huang  Y, Feng  Z.  Biases introduced by choosing controls to match risk factors of cases in biomarker research.  Clin Chem. 2012;58(8):1242-1251.PubMedGoogle ScholarCrossref
32.
Usher-Smith  JA, Emery  J, Kassianos  AP, Walter  FM.  Risk prediction models for melanoma: a systematic review.  Cancer Epidemiol Biomarkers Prev. 2014;23(8):1450-1463.PubMedGoogle ScholarCrossref
33.
Fortes  C, Mastroeni  S, Bakos  L,  et al.  Identifying individuals at high risk of melanoma: a simple tool.  Eur J Cancer Prev. 2010;19(5):393-400.PubMedGoogle ScholarCrossref
34.
Cust  AE, Goumas  C, Vuong  K,  et al.  MC1R genotype as a predictor of early-onset melanoma, compared with self-reported and physician-measured traditional risk factors: an Australian case-control-family study.  BMC Cancer. 2013;13:406.PubMedGoogle ScholarCrossref
35.
Cust  AE, Pickles  KM, Goumas  C,  et al.  Accuracy of self-reported nevus and pigmentation phenotype compared with clinical assessment in a population-based study of young Australian adults.  Cancer Epidemiol Biomarkers Prev. 2015;24(4):736-743.PubMedGoogle ScholarCrossref
36.
Anothaisintawee  T, Teerawattananon  Y, Wiratkapun  C, Kasamesup  V, Thakkinstian  A.  Risk prediction models of breast cancer: a systematic review of model performances.  Breast Cancer Res Treat. 2012;133(1):1-10.PubMedGoogle ScholarCrossref
37.
Win  AK, Macinnis  RJ, Hopper  JL, Jenkins  MA.  Risk prediction models for colorectal cancer: a review.  Cancer Epidemiol Biomarkers Prev. 2012;21(3):398-410.PubMedGoogle ScholarCrossref
38.
Usher-Smith  JA, Walter  FM, Emery  JD, Win  AK, Griffin  SJ.  Risk prediction models for colorectal cancer: a systematic review.  Cancer Prev Res (Phila). 2016;9(1):13-26.http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=24895414&dopt=AbstractPubMedGoogle ScholarCrossref
39.
Spitz  MR, Etzel  CJ, Dong  Q,  et al.  An expanded risk prediction model for lung cancer.  Cancer Prev Res (Phila). 2008;1(4):250-254.PubMedGoogle ScholarCrossref
40.
Lughezzani  G, Briganti  A, Karakiewicz  PI,  et al.  Predictive and prognostic models in radical prostatectomy candidates: a critical analysis of the literature.  Eur Urol. 2010;58(5):687-700.PubMedGoogle ScholarCrossref
41.
Siontis  GC, Tzoulaki  I, Castaldi  PJ, Ioannidis  JP.  External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination.  J Clin Epidemiol. 2015;68(1):25-34.PubMedGoogle ScholarCrossref
42.
Kramer  AA, Zimmerman  JE.  Assessing the calibration of mortality benchmarks in critical care: the Hosmer-Lemeshow test revisited.  Crit Care Med. 2007;35(9):2052-2056.PubMedGoogle ScholarCrossref
43.
Davies  JR, Chang  YM, Bishop  DT,  et al.  Development and validation of a melanoma risk score based on pooled data from 16 case-control studies.  Cancer Epidemiol Biomarkers Prev. 2015;24(5):817-824.PubMedGoogle ScholarCrossref
44.
Collins  GS, Altman  DG.  Identifying patients with undetected colorectal cancer: an independent validation of QCancer (colorectal).  Br J Cancer. 2012;107(2):260-265.PubMedGoogle ScholarCrossref
45.
Li  K, Hüsing  A, Sookthai  D,  et al.  Selecting high-risk individuals for lung cancer screening—a prospective evaluation of existing risk models and eligibility criteria in the German EPIC cohort.  Cancer Prev Res (Phila). 2015;8(9):777-785.PubMedGoogle ScholarCrossref
46.
Foley  RW, Gorman  L, Sharifi  N,  et al.  Improving multivariable prostate cancer risk assessment using the Prostate Health Index.  BJU Int. 2016;117(3):409-417.PubMedGoogle ScholarCrossref
Original Investigation
August 2016

Development and External Validation of a Melanoma Risk Prediction Model Based on Self-assessed Risk Factors

Author Affiliations
  • 1Cancer Epidemiology and Prevention Research, Sydney School of Public Health, University of Sydney, Sydney, Australia
  • 2School of Public Health and Community Medicine, University of New South Wales, Sydney, Australia
  • 3Department of Community Medicine, Faculty of Health Sciences, University of Tromsø, Arctic University of Norway, Tromsø, Norway
  • 4Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden
  • 5Department of Research, Cancer Registry of Norway, Institute of Population-Based Cancer Research, Oslo, Norway
  • 6Genetic Epidemiology Group, Folkhälsan Research Center, Helsinki, Finland
  • 7Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, Massachusetts
  • 8Oslo Centre for Biostatistics and Epidemiology, Department of Biostatistics, Institute of Basic Medical Sciences, University of Oslo, Oslo, Norway
  • 9Leeds Institute of Cancer and Pathology, Faculty of Medicine and Health, Leeds University, Leeds, United Kingdom
  • 10Population Health Department, QIMR Berghofer Medical Research Institute, Brisbane, Australia
  • 11Centre for Epidemiology and Biostatistics, Melbourne School of Population and Global Health, University of Melbourne, Melbourne, Australia
  • 12Centre for Cancer Research, Westmead Institute for Medical Research, University of Sydney, Westmead, Australia
  • 13Melanoma Institute Australia, University of Sydney, North Sydney, Australia
  • 14Sydney School of Public Health, University of Sydney, Sydney, Australia
JAMA Dermatol. 2016;152(8):889-896. doi:10.1001/jamadermatol.2016.0939
Abstract

Importance  Identifying individuals at high risk of melanoma can optimize primary and secondary prevention strategies.

Objective  To develop and externally validate a risk prediction model for incident first-primary cutaneous melanoma using self-assessed risk factors.

Design, Setting, and Participants  We used unconditional logistic regression to develop a multivariable risk prediction model. Relative risk estimates from the model were combined with Australian melanoma incidence and competing mortality rates to obtain absolute risk estimates. A risk prediction model was developed using the Australian Melanoma Family Study (629 cases and 535 controls) and externally validated using 4 independent population-based studies: the Western Australia Melanoma Study (511 case-control pairs), Leeds Melanoma Case-Control Study (960 cases and 513 controls), Epigene-QSkin Study (44 544, of which 766 with melanoma), and Swedish Women’s Lifestyle and Health Cohort Study (49 259 women, of which 273 had melanoma).

Main Outcomes and Measures  We validated model performance internally and externally by assessing discrimination using the area under the receiver operating curve (AUC). Additionally, using the Swedish Women’s Lifestyle and Health Cohort Study, we assessed model calibration and clinical usefulness.

Results  The risk prediction model included hair color, nevus density, first-degree family history of melanoma, previous nonmelanoma skin cancer, and lifetime sunbed use. On internal validation, the AUC was 0.70 (95% CI, 0.67-0.73). On external validation, the AUC was 0.66 (95% CI, 0.63-0.69) in the Western Australia Melanoma Study, 0.67 (95% CI, 0.65-0.70) in the Leeds Melanoma Case-Control Study, 0.64 (95% CI, 0.62-0.66) in the Epigene-QSkin Study, and 0.63 (95% CI, 0.60-0.67) in the Swedish Women’s Lifestyle and Health Cohort Study. Model calibration showed close agreement between predicted and observed numbers of incident melanomas across all deciles of predicted risk. In the external validation setting, there was higher net benefit when using the risk prediction model to classify individuals as high risk compared with classifying all individuals as high risk.

Conclusions and Relevance  The melanoma risk prediction model performs well and may be useful in prevention interventions reliant on a risk assessment using self-assessed risk factors.

Introduction

Melanoma incidence has been increasing in predominantly fair-skinned populations, with Australia having the world’s highest rates.1 Primary prevention measures, based on sun protection, are a priority for reducing the melanoma burden.2 Risk prediction models have been proposed as a more accurate and informative way of communicating risk3 and may lead to better preventive behaviors among those at high risk. Additionally, risk stratification may assist in planning intervention trials and targeting population prevention interventions.4

Most published melanoma risk prediction models have limited reporting of methods and results, and few have been externally validated.5,6 External validation evaluates model performance using independent data and is important to perform before routine clinical use.7 We aimed to develop a model for incident first-primary cutaneous melanoma based on self-assessed risk factors from the Australian Melanoma Family Study,8 and to externally validate the model in the Western Australian Melanoma Study,9 the Leeds Melanoma Case-Control Study,10,11 the Epigene-QSkin Study,12,13 and the Swedish Women’s Lifestyle and Health Cohort Study.14,15

Box Section Ref ID

Key Points

  • Question How well does a melanoma risk prediction model, developed using self-reported risk factors, predict incident first-primary cutaneous melanoma in the development dataset and 4 independent populations?

  • Findings This study found that the melanoma risk prediction model, which included hair color, nevus density, first-degree family history of melanoma, previous nonmelanoma skin cancer, and lifetime sunbed use, predicts well on measures of discrimination, calibration, and net benefit.

  • Meaning The melanoma risk prediction model may be useful in prevention interventions reliant on risk assessment using self-reported risk factors.

Methods

Studies were approved by the Human Research Ethic Committees at the University of Sydney, UK Multi-Centre (MREC), Patient Advisory Group (PIAG), QIMR Berghofer Medical Research Institute, and Swedish Data Inspection Board. Informed verbal consent was obtained from the Western Australia Melanoma Study participants, and written consent was obtained from all other participants.

Participants

The Australian Melanoma Family Study is a population-based, case-control-family study with 629 incident first-primary cutaneous melanoma cases, 240 controls, and 295 spouse or friend controls from Brisbane, Sydney, and Melbourne, Australia.8 Cases were identified from state cancer registries and diagnosed between July 2000 and December 2002 at ages 18 to 39 years; participation was 54%. Controls were identified from the electoral roll (registration to vote is compulsory in Australia) and were frequency matched to cases by city, age, and sex; participation was 23%. In addition, cases were asked to nominate a spouse or friend as a potential control participant; participation was 80%. Data were collected using self-administered and interviewer-administered questionnaires.

The Western Australia Melanoma Study is a population-based study with 511 case-control pairs.9 Cases were identified from clinicians and pathology registers, and diagnosed between January 1980 and November 1981 at ages 10 to 80 years; participation was 76%. Controls were selected from the electoral roll and were frequency matched to cases by electoral subdivision, age, and sex; participation was 69%. Nurses collected data by administering a questionnaire and recording the number of raised nevi on the arm.

The Leeds Melanoma Case-Control Study is a population-based case-control study with 960 melanoma cases and 513 controls from Yorkshire, United Kingdom.10,11 Cases were identified from clinicians, pathology registers, and cancer registries, and diagnosed between September 2000 and December 2005 at ages 18 to 76 years; participation was 67%. Controls were selected from the cases’ general practice (usually the practice nearest to their home residence) and were frequency matched to cases by age and sex; participation was 55%. Data were collected using self-administered and telephone-administered questionnaires.

The Epigene-QSkin Study comprised harmonized variables for 766 melanoma cases from the Epigene case-case study12 and 43 778 participants without melanoma from the QSkin Cohort Study.13 Cases were identified from pathology registers from the Brisbane region, Australia, and diagnosed between April 2007 and September 2010 at ages 18 to 79 years; participation was 52%. QSkin Cohort Study participants were randomly identified from the electoral roll, at ages 40 to 69 years and living in Queensland, Australia, between November 2010 and November 2011. Data were collected using self-administered questionnaires.

The Swedish Women’s Lifestyle and Health Cohort Study is a prospective study with 49 259 women.14,15 Participants were randomly identified from the Central Population Register at Statistics Sweden, at ages 30 to 50 years and living in the Uppsala Health Care Region in 1991 or 1992. Linkage of the cohort study to the national cancer registry to December 31, 2011, identified 273 women with incident first-primary melanoma. Data were collected using self-administered questionnaires. The Norwegian twin cohort to the Swedish Women’s Lifestyle and Health Cohort Study was not included in the validation analyses because information on family history of melanoma was not collected.15

Model Development

We used unconditional logistic regression to derive a multivariable risk prediction model using the Australian Melanoma Family Study. The following self-assessed melanoma risk factors were used as candidate predictors: age, sex, city of recruitment, country of birth, ethnicity, skin color, eye color, natural hair color at age 18 years, skin response to sunlight, nevus density (based on 4-level pictogram) (eFigure 1 in the Supplement), freckle density (based on 6-level pictogram), personal history of nonmelanoma skin cancer, first-degree family history of melanoma, blistering sunburn frequency (childhood and lifetime), sunbed use, and sunscreen use.1620 We adjusted for age, sex, and city of recruitment by keeping these variables in each step. Variables with P >.05 were removed using backward selection. Continuous variables were analyzed as a linear function, as P values for nonlinearity were greater than .05, and then categorized in the final model. Effect modification was tested by adding terms for the interaction between each variable and each other variable included in the final model, 1 interaction term at a time. We used multiple imputation by chained equations with 10 imputed data sets to impute missing values.21

Age (a) from 0 to 85 years was divided into 5-year age groups j (j = 1, 2,…, 16, 17; [0,τ1), [τ12),…,[τ16,τ17)). Lifetime (to 85 years of age) and 20-year absolute risks (P) for an individual aged a with relative risk r were estimated using the method of Gail et al22 by (1) calculating the attributable fraction (AF) from the distribution of relative risk among the cases,23 (2) multiplying the Australian age-specific melanoma incidence rates (h1*) by (1 − AF) to give h1, and (3) using h2, the mortality rates from causes other than melanoma between 2007 and 2009 (eTable 1 in the Supplement), as shown in the following formula:

Image description not available.where in the summation, the smallest j value satisfies τj-1= a, the largest j value satisfies τj = a+τ, and the value of τ is the time interval over which we calculate the absolute risk, for example, to calculate 20-year absolute risk τ = 20. S1, the probability of remaining melanoma free up to age τj, was estimated by S1(τj) = S1(τ j-1)exp(−5h1jrj), where S1(0) = 1. S2, the probability of surviving competing risk up to age τj, was estimated by S2(τj) = S2(τj-1)exp(−5h2), where S2(0) = 1.

Model Performance and Validation

We evaluated model performance in the development data set (internal validation) and externally using 4 independent validation data sets by assessing discrimination (the ability to distinguish between those with and without melanoma) using the area under the receiver operating characteristic curve (AUC), with values ranging from 0.5 (no better than chance) to 1 (perfect discrimination).24(p261) Additionally, we assessed calibration and clinical usefulness in the Swedish Women’s Lifestyle and Health Cohort Study15,25 over 20 years of follow-up, by examining the calibration plot, calibration slope, calibration-in-the-large, net benefit, and decision curve (obtained from plotting the net benefit at different absolute risk thresholds). The calibration plot depicts the observed and predicted numbers of incident melanomas by deciles of predicted risk.26 The calibration-in-the-large (intercept) and calibration slope (slope) are obtained from plotting the log odds of predictions as the predictor, with an intercept of 0 and slope of 1 indicating perfect calibration.26 Net benefit was calculated by weighing the true-positive against the false-positive classifications at different absolute risk thresholds; the relative weight of the true-positive to false-positive classifications is determined by the absolute risk threshold, with higher net benefit indicating greater clinical usefulness.27 We used bootstrapping procedures with 1000 repetitions to estimate 95% confidence intervals.

Variables in the validation data sets were harmonized to those in the risk prediction model. The number of raised nevi on the arms (in the Western Australia Melanoma Study) and large asymmetric nevi on lower limbs (in the Swedish Women’s Lifestyle and Health Cohort Study) were matched to the approximate nevus counts shown on the Australian Melanoma Family Study pictograms (eFigure 1 in the Supplement). Data on sunbed use were not collected in the Epigene Study; thus, we assumed that none of its participants used sunbeds. Lifetime (to 85 years of age) and 20-year absolute risks were estimated using the method of Gail et al.22 We excluded validation study participants who had missing values for any of the predictor variables.28 The total participants included in the analyses and missing rates are shown in eTable 2 in the Supplement.

Data were analyzed using Stata, version 12 (for model development), and SAS, version 9.3 (for model validation), with 2-sided P values. Statistical significance was inferred at P < .05, except for interaction terms, for which we used a more stringent P < .01 to allow for multiple testing.29 We report methods and results in accordance with the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis statement.30

Results

Quiz Ref IDThe final melanoma risk prediction model included hair color, nevus density, first-degree family history of melanoma, previous nonmelanoma skin cancer, and sunbed use, with red hair color and nevus density the strongest predictors of risk (Table). Sunbed use was associated with melanoma when analyzed as a linear function (P = .04), but the P value was .20 when categorized. There were no significant interactions between pairs of variables in the final model.

Relative risk estimates for the model predictors were generally similar in the development and validation data sets (Table). However, the relative risks for red hair color in the Western Australian Melanoma Study and for personal history of nonmelanoma skin cancer in the Leeds Melanoma Case-Control Study and Epigene-QSkin Study were lower than in the development model. Distributions of the predictor variables in the development and validation data sets are shown in eTable 2 in the Supplement.

On internal validation, the AUC was 0.70 (95% CI, 0.67-0.73) in the development data set. On external validation, the AUC was 0.66 (95% CI, 0.63-0.69) in the Western Australia Melanoma Study, 0.67 (95% CI, 0.65-0.70) in the Leeds Melanoma Case-Control Study, 0.64 (95% CI, 0.62-0.66) in the Epigene-QSkin Study, and 0.63 (95% CI, 0.60-0.67) in the Swedish Women’s Lifestyle and Health Cohort Study. Quiz Ref IDThe calibration plot showed close agreement between predicted and observed numbers of incident melanomas across all deciles of predicted risk over 20 years of follow-up (Figure 1). In the lowest decile of predicted risk, for example, the model predicted a mean of 11.89 melanomas and 11 melanomas were observed. Calibration-in-the-large was −0.20 (95% CI, −0.21 to −0.19), and the calibration slope was 0.79 (95% CI, 0.64 to 0.95), indicating that the model might give an overestimate of risk.

Figure 2 compares the decision curves from classifying individuals as high risk using the risk prediction model, classifying all individuals as high risk, and classifying all individuals as low risk (horizontal line at 0) over 20 years of follow-up. Quiz Ref IDClassifying individuals as high risk using the model had higher net benefit compared with classifying all individuals as high risk across all 20-year absolute risk thresholds. Classifying individuals as high risk using the model also had higher net benefit compared with classifying all individuals as low risk for 20-year absolute risk thresholds of 1% or less. However, for 20-year absolute risk thresholds above 1%, classifying all individuals as low risk had higher net benefit than classifying individuals as high risk using the model. To demonstrate, if the absolute risk threshold for classifying individuals as high risk and warranting prevention intervention is 1% and 100 000 individuals are observed over 20 years, classifying individuals as high risk using the risk prediction model would identify 161 individuals expected to receive a diagnosis of melanoma (true positive) and 15 265 individuals without melanoma (false positive) as high risk. This has a positive net benefit of 0.00007, calculated as follows:

Image description not available.or [161 – (1/99 × 15 265)]/100 000, as the benefit of true-positive classifications outweighs the harms of false-positive classifications at this absolute risk threshold. In comparison, classifying all individuals as high risk would identify 548 individuals expected to receive a diagnosis of melanoma (true positive) and 99 452 individuals without melanoma (false positive) as high risk. This has a negative net benefit of –0.00456, calculated as follows: [548 − (1/99 × 99 452)]/100 000, as the benefit of true-positive classifications is outweighed by the harms of false-positive classifications at this absolute risk threshold. Classifying all individuals as low risk has a net benefit of 0 because there are no true-positive classifications (no benefits) and no false-positive classifications (no harms).

In matched case-control studies, the distribution of risk factors among controls is more similar to the cases than to the general population.31 We conducted sensitivity analyses to reweight the age and sex distribution of the Western Australia Melanoma and Leeds Melanoma Case-Control studies’ controls to the Western Australian and Leeds population, respectively. This reweighting procedure did not change the AUC in the Western Australia Study and reduced the AUC to 0.60 (95% CI, 0.57-0.62) in the Leeds Melanoma Case-Control Study. This may be due to the small number of controls (and hence large weights) among the youngest age strata in the Leeds Melanoma Case-Control Study (eMethods 1 and eTables 3-5 in the Supplement). Melanoma incidence rates in Sweden have been increasing but are lower than Australian rates.1 Sensitivity analyses to recalibrate the risk prediction model using Swedish melanoma incidence and mortality rates from 2009 to 2011 to estimate the 20-year absolute risk showed little change in model calibration (eMethods 2, eTable 6, and eFigure 2 in the Supplement). However, when we used the lower Swedish melanoma incidence rates from 1991 to 2011 to estimate the 20-year absolute risk, calibration was poorer (eTable 7 and eFigure 3 in the Supplement).

Discussion

Quiz Ref IDThis melanoma risk prediction model was developed for use in clinical and population interventions reliant on use of self-assessed risk factors. The model included hair color, nevus density, first-degree family history of melanoma, previous nonmelanoma skin cancer, and lifetime sunbed use. The model showed good discrimination on internal validation (AUC = 0.70 [95% CI, 0.67-0.73]), with lower discrimination on external validation (AUCs ranging from 0.63 to 0.67 across the 4 validation data sets). The model was well calibrated and had higher net benefit compared with classifying all individuals as high risk across all 20-year absolute risk thresholds.

For discrimination, the model compared well to risk prediction models for melanoma and other cancers. Systematic reviews of melanoma risk prediction models have shown AUCs ranging from 0.62 to 0.86 on internal validation.5,32 In one of the few models with external validation, Fortes and colleagues33 reported an AUC of 0.79. Discriminative performance tends to be higher when based on clinically measured nevi,34 such as in the model of Fortes and colleagues,33 probably because self-reports tend to underestimate nevus counts in comparison with clinical assessment.35 The AUCs of risk prediction models for other cancers ranged from 0.53 to 0.66 for breast cancer,36 0.62 to 0.75 for colorectal cancer,37,38 0.67 to 0.73 for lung cancer,39 and 0.52 to 0.93 for prostate cancer,40 with poorer discrimination on external validation.41

A strength of our study was the use of calibration and newer model performance measures: net benefit and decision curve analyses using an independent cohort study. Previous melanoma risk prediction models that reported calibration used the Hosmer-Lemeshow test, which is sensitive to sample size42 and has low power to detect overfitting of predictor effects.24 Presenting the calibration plot, calibration-in-the-large, and calibration slope, as we have done, is the preferred method.7,26 To our knowledge, no other melanoma prediction model has evaluated model performance using net benefit and decision curve analyses.5,32,43 A few prediction models for other cancers have found, as we did, that using the model to classify individuals at high risk using reasonably low absolute risk thresholds had higher net benefit compared with classifying all individuals as high risk.4446

Based on net benefit analyses, our model is most useful at classifying individuals as high risk and warranting risk-based interventions if the 20-year absolute risk threshold is 1% or less. In the Australian Melanoma Family Study,8 our development data set, 58% of participants had a model-estimated 20-year absolute risk of 1% or less. Examples of Australian Melanoma Family Study participants with a model-estimated 20-year absolute risk of 1% include (1) a man aged 38 years with light brown hair, some nevi, no first-degree family history of melanoma, no personal history of nonmelanoma skin cancer, and 1 to 10 episodes of prior sunbed use; and (2) a woman aged 32 with light brown hair, many nevi, no first-degree melanoma family history, no personal history of nonmelanoma skin cancer, and no sunbed use. For 20-year absolute risk thresholds set at 1% or less, using the model to classify individuals as high risk for risk-based interventions would be better than either assuming that everyone is high risk (intervening) or assuming that everyone is low risk (not intervening). However, for 20-year absolute risk thresholds set above 1%, the model would be no better than assuming that everyone is low risk (not intervening).

The choice of a risk threshold for intervention will likely vary depending on the efficacy and potential harms associated with the intervention and subsequent management for individuals classified as high risk. If the intervention and subsequent management has high efficacy and low potential harms, then the risk threshold for intervention will be low. In contrast, if the intervention and subsequent management has low efficacy and high potential harms, then the risk threshold for intervention will be high.

Direct comparison with previous melanoma risk prediction models and validation studies is difficult because of differences in the study designs, predictor variable definitions, data handling, and reporting. It is a potential limitation that our model was developed using a data set in which all melanoma cases were diagnosed at younger than 40 years (ie, early onset). Although there is some evidence that the strength of melanoma risk factors may vary with age,19 our model performed well on external validation in older populations. Due to few cohort studies having melanoma risk factor data available for external validation, we were only able to evaluate model calibration and net benefit in women in the Swedish Women’s Lifestyle and Health Cohort Study over 20 years of follow-up. Predictor variables in the validation data sets were harmonized as closely as possible to those in the development model, but sunbed use was not collected on all data sets. In assuming no sunbed use in the Epigene Study, and sunlamps to infer sunbed use in the Western Australia Melanoma Study, the discriminative performance of our model is probably an underestimate. Other potential limitations of our study include participation bias and inaccuracy of self-reported risk factors. The discriminative performance of our model would probably have been higher if based on clinically measured nevi,45 but clinical measurement is more expensive, more time-consuming, and less accessible than self-assessment.

Conclusions

Quiz Ref IDThis risk prediction model developed using self-assessed risk factors demonstrated good discrimination and calibration, and performed satisfactorily on external validation. It could be used to inform individuals of their risk of developing melanoma and to stratify them into risk categories using 20-year absolute risk thresholds of 1% or less for targeted primary and secondary prevention interventions. Feasibility, impact on care, and cost-effectiveness should be prospectively evaluated before a model such as ours is put into routine use in clinical practice.

Back to top
Article Information

Group Information: The Australian Melanoma Family Study Investigators are listed at the end of the article.

Accepted for Publication: March 10, 2016.

Corresponding Author: Kylie Vuong, MBBS, MIPH, FRACGP, Cancer Epidemiology and Prevention Research, Sydney School of Public Health, University of Sydney, New South Wales 2006, Australia (kylie.vuong@sydney.edu.au).

Group Information: Australian Melanoma Family Study Investigators: Joanne F. Aitken, PhD, Cancer Council Queensland; Menzies Health Institute Queensland, Griffith University; School of Public Health and Social Work, Queensland University of Technology; and Institute for Resilient Regions, University of Southern Queensland, Brisbane, Australia; Graham G. Giles, PhD, Department of Epidemiology and Preventive Medicine, School of Public Health and Preventive Medicine, Monash University; Centre for Epidemiology and Biostatistics, Melbourne School of Population and Global Health, University of Melbourne; and Cancer Epidemiology Centre, Cancer Council Victoria, Melbourne, Australia; Richard F. Kefford, PhD, Macquarie University Health Sciences Centre, Sydney; Melanoma Institute Australia, University of Sydney, North Sydney; and Crown Princess Mary Cancer Centre, Westmead Hospital, Westmead, Australia; Bruce K. Armstrong, MBBS(Hons), PhD, FAFPHM, Cancer Epidemiology and Prevention Research, Sydney School of Public Health, University of Sydney, Sydney, Australia; John L. Hopper, PhD, Centre for Epidemiology and Biostatistics, Melbourne School of Population and Global Health, University of Melbourne, Melbourne, Australia; Helen Schmid, MPH, Centre for Cancer Research, Westmead Institute for Medical Research, University of Sydney, Westmead; and Melanoma Institute Australia, University of Sydney, North Sydney, Australia; Mark A. Jenkins, PhD, Centre for Epidemiology and Biostatistics, Melbourne School of Population and Global Health, University of Melbourne, Melbourne, Australia; Anne E. Cust, MPH(Hons), PhD, Cancer Epidemiology and Prevention Research, Sydney School of Public Health, University of Sydney, Sydney; and Melanoma Institute Australia, University of Sydney, North Sydney, Australia; and Graham J. Mann, PhD, Centre for Cancer Research, Westmead Institute for Medical Research, University of Sydney, Westmead; and Melanoma Institute Australia, University of Sydney, North Sydney, Australia.

Published Online: June 8, 2016. doi:10.1001/jamadermatol.2016.0939.

Author Contributions: Drs Vuong and McGeechan had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Drs Cust and McGeechan each contributed equally to this work.

Study concept and design: Vuong, Armstrong, Weiderpass, Hopper, Mann, Cust.

Acquisition, analysis, or interpretation of data: Vuong, Armstrong, Weiderpass, Lund, Adami, Veierod, Barrett, Davies, Bishop, Whiteman, Olsen, Hopper, Mann, Cust, McGeechan.

Drafting of the manuscript: Vuong, Armstrong, Weiderpass, Hopper.

Critical revision of the manuscript for important intellectual content: Vuong, Armstrong, Weiderpass, Lund, Adami, Veierod, Barrett, Davies, Bishop, Whiteman, Olsen, Hopper, Mann, Cust, McGeechan.

Statistical analysis: Vuong, Weiderpass, Barrett, Davies, Olsen, Hopper.

Obtained funding: Vuong, Armstrong, Weiderpass, Bishop, Hopper, Mann, Cust.

Administrative, technical, or material support: Vuong, Weiderpass, Lund, Adami, Veierod, Whiteman, Hopper, Mann, Cust.

Study supervision: Armstrong, Weiderpass, Barrett, Hopper, Mann, Cust, McGeechan.

Conflict of Interest Disclosures: None reported.

Funding/Support: Dr Vuong was supported by a University of Sydney Postgraduate Scholarship in Cancer Epidemiology and a Sydney Catalyst Top-Up Research Scholar Award. Dr Cust was supported by fellowships from the Cancer Institute New South Wales (10/ECF/2-06) and the Australian National Health and Medical Research Council (NHMRC) (1063593). The Australian Melanoma Family Study received funding from the NHMRC (project grants 107359, 211172 and Program Grant 402761 to G.J.M. and R.F.K.); project grants from the Cancer Council of New South Wales (77/00, 06/10), Cancer Council of Victoria, and Cancer Council of Queensland (371); and the US National Institutes of Health (via RO1 grant CA-83115-01A2 to the international Melanoma Genetics Consortium, GenoMEL). The Swedish Women’s Lifestyle and Health study is supported by a grant from the Swedish Research Council (Vetenskapsrådet) (K2012-69X-22062-01-3).

Role of the Funder/Sponsor: The sponsors had no role in the design and conduct of the study; in the collection, analysis, and interpretation of data; or in the preparation, review, or approval of the manuscript, or decision to submit the manuscript for publication.

References
1.
Erdmann  F, Lortet-Tieulent  J, Schüz  J,  et al.  International trends in the incidence of malignant melanoma 1953-2008—are recent generations at higher or lower risk?  Int J Cancer. 2013;132(2):385-400.PubMedGoogle ScholarCrossref
2.
Stanton  WR, Janda  M, Baade  PD, Anderson  P.  Primary prevention of skin cancer: a review of sun protection in Australia and internationally.  Health Promot Int. 2004;19(3):369-378.PubMedGoogle ScholarCrossref
3.
Ahmed  H, Naik  G, Willoughby  H, Edwards  AG.  Communicating risk.  BMJ. 2012;344:e3996.PubMedGoogle ScholarCrossref
4.
Freedman  AN, Seminara  D, Gail  MH,  et al.  Cancer risk prediction models: a workshop on development, evaluation, and application.  J Natl Cancer Inst. 2005;97(10):715-723.PubMedGoogle ScholarCrossref
5.
Vuong  K, McGeechan  K, Armstrong  BK, Cust  AE.  Risk prediction models for incident primary cutaneous melanoma: a systematic review.  JAMA Dermatol. 2014;150(4):434-444.PubMedGoogle ScholarCrossref
6.
Olsen  CM, Neale  RE, Green  AC, Webb  PM, Whiteman  DC; QSkin Study; Epigene Study.  Independent validation of six melanoma risk prediction models.  J Invest Dermatol. 2015;135(5):1377-1384.PubMedGoogle ScholarCrossref
7.
Collins  GS, de Groot  JA, Dutton  S,  et al.  External validation of multivariable prediction models: a systematic review of methodological conduct and reporting.  BMC Med Res Methodol. 2014;14:40.PubMedGoogle ScholarCrossref
8.
Cust  AE, Schmid  H, Maskiell  JA,  et al.  Population-based, case-control-family design to investigate genetic and environmental influences on melanoma risk: Australian Melanoma Family Study.  Am J Epidemiol. 2009;170(12):1541-1554.PubMedGoogle ScholarCrossref
9.
English  DR, Armstrong  BK.  Identifying people at high risk of cutaneous malignant melanoma: results from a case-control study in Western Australia.  Br Med J (Clin Res Ed). 1988;296(6632):1285-1288.PubMedGoogle ScholarCrossref
10.
Newton-Bishop  JA, Chang  YM, Iles  MM,  et al.  Melanocytic nevi, nevus genes, and melanoma risk in a large case-control study in the United Kingdom.  Cancer Epidemiol Biomarkers Prev. 2010;19(8):2043-2054.PubMedGoogle ScholarCrossref
11.
Newton-Bishop  JA, Chang  YM, Elliott  F,  et al.  Relationship between sun exposure and melanoma risk for tumours in different body sites in a large case-control study in a temperate climate.  Eur J Cancer. 2011;47(5):732-741.PubMedGoogle ScholarCrossref
12.
Kvaskoff  M, Pandeya  N, Green  AC,  et al.  Site-specific determinants of cutaneous melanoma: a case-case comparison of patients with tumors arising on the head or trunk.  Cancer Epidemiol Biomarkers Prev. 2013;22(12):2222-2231.PubMedGoogle ScholarCrossref
13.
Olsen  CM, Green  AC, Neale  RE,  et al; QSkin Study.  Cohort profile: the QSkin Sun and Health Study.  Int J Epidemiol. 2012;41(4):929-929i.PubMedGoogle ScholarCrossref
14.
Roswall  N, Sandin  S, Adami  HO, Weiderpass  E.  Cohort profile: the Swedish Women’s Lifestyle and Health cohort [published online June 10, 2015].  Int J Epidemiol. doi:10.1093/ije/dyv089.PubMedGoogle Scholar
15.
Veierød  MB, Adami  HO, Lund  E, Armstrong  BK, Weiderpass  E.  Sun and solarium exposure and melanoma risk: effects of age, pigmentary characteristics, and nevi.  Cancer Epidemiol Biomarkers Prev. 2010;19(1):111-120.PubMedGoogle ScholarCrossref
16.
Gandini  S, Sera  F, Cattaruzza  MS,  et al.  Meta-analysis of risk factors for cutaneous melanoma: I. common and atypical naevi.  Eur J Cancer. 2005;41(1):28-44.PubMedGoogle ScholarCrossref
17.
Gandini  S, Sera  F, Cattaruzza  MS,  et al.  Meta-analysis of risk factors for cutaneous melanoma: II. sun exposure.  Eur J Cancer. 2005;41(1):45-60.PubMedGoogle ScholarCrossref
18.
Gandini  S, Sera  F, Cattaruzza  MS,  et al.  Meta-analysis of risk factors for cutaneous melanoma: III. family history, actinic damage and phenotypic factors.  Eur J Cancer. 2005;41(14):2040-2059.PubMedGoogle ScholarCrossref
19.
Cust  AE, Armstrong  BK, Goumas  C,  et al.  Sunbed use during adolescence and early adulthood is associated with increased risk of early-onset melanoma.  Int J Cancer. 2011;128(10):2425-2435.PubMedGoogle ScholarCrossref
20.
Chang  YM, Barrett  JH, Bishop  DT,  et al.  Sun exposure and melanoma risk at different latitudes: a pooled analysis of 5700 cases and 7216 controls.  Int J Epidemiol. 2009;38(3):814-830.PubMedGoogle ScholarCrossref
21.
Vergouwe  Y, Royston  P, Moons  KG, Altman  DG.  Development and validation of a prediction model with missing predictor data: a practical approach.  J Clin Epidemiol. 2010;63(2):205-214.PubMedGoogle ScholarCrossref
22.
Gail  MH, Brinton  LA, Byar  DP,  et al.  Projecting individualized probabilities of developing breast cancer for white females who are being examined annually.  J Natl Cancer Inst. 1989;81(24):1879-1886.PubMedGoogle ScholarCrossref
23.
Bruzzi  P, Green  SB, Byar  DP, Brinton  LA, Schairer  C.  Estimating the population attributable risk for multiple risk factors using case-control data.  Am J Epidemiol. 1985;122(5):904-914.PubMedGoogle Scholar
24.
Steyerberg  EW.  Clinical Prediction Models. New York, NY: Springer; 2010.
25.
Veierød  MB, Weiderpass  E, Thörn  M,  et al.  A prospective study of pigmentation, sun exposure, and risk of cutaneous malignant melanoma in women.  J Natl Cancer Inst. 2003;95(20):1530-1538.PubMedGoogle ScholarCrossref
26.
Steyerberg  EW, Vergouwe  Y.  Towards better clinical prediction models: seven steps for development and an ABCD for validation.  Eur Heart J. 2014;35(29):1925-1931.PubMedGoogle ScholarCrossref
27.
Vickers  AJ, Elkin  EB.  Decision curve analysis: a novel method for evaluating prediction models.  Med Decis Making. 2006;26(6):565-574.PubMedGoogle ScholarCrossref
28.
Royston  P, Moons  KG, Altman  DG, Vergouwe  Y.  Prognosis and prognostic research: developing a prognostic model.  BMJ. 2009;338:b604.PubMedGoogle ScholarCrossref
29.
Bender  R, Lange  S.  Adjusting for multiple testing—when and how?  J Clin Epidemiol. 2001;54(4):343-349.PubMedGoogle ScholarCrossref
30.
Collins  GS, Reitsma  JB, Altman  DG, Moons  KG.  Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement.  BMJ. 2015;350:g7594.PubMedGoogle ScholarCrossref
31.
Pepe  MS, Fan  J, Seymour  CW, Li  C, Huang  Y, Feng  Z.  Biases introduced by choosing controls to match risk factors of cases in biomarker research.  Clin Chem. 2012;58(8):1242-1251.PubMedGoogle ScholarCrossref
32.
Usher-Smith  JA, Emery  J, Kassianos  AP, Walter  FM.  Risk prediction models for melanoma: a systematic review.  Cancer Epidemiol Biomarkers Prev. 2014;23(8):1450-1463.PubMedGoogle ScholarCrossref
33.
Fortes  C, Mastroeni  S, Bakos  L,  et al.  Identifying individuals at high risk of melanoma: a simple tool.  Eur J Cancer Prev. 2010;19(5):393-400.PubMedGoogle ScholarCrossref
34.
Cust  AE, Goumas  C, Vuong  K,  et al.  MC1R genotype as a predictor of early-onset melanoma, compared with self-reported and physician-measured traditional risk factors: an Australian case-control-family study.  BMC Cancer. 2013;13:406.PubMedGoogle ScholarCrossref
35.
Cust  AE, Pickles  KM, Goumas  C,  et al.  Accuracy of self-reported nevus and pigmentation phenotype compared with clinical assessment in a population-based study of young Australian adults.  Cancer Epidemiol Biomarkers Prev. 2015;24(4):736-743.PubMedGoogle ScholarCrossref
36.
Anothaisintawee  T, Teerawattananon  Y, Wiratkapun  C, Kasamesup  V, Thakkinstian  A.  Risk prediction models of breast cancer: a systematic review of model performances.  Breast Cancer Res Treat. 2012;133(1):1-10.PubMedGoogle ScholarCrossref
37.
Win  AK, Macinnis  RJ, Hopper  JL, Jenkins  MA.  Risk prediction models for colorectal cancer: a review.  Cancer Epidemiol Biomarkers Prev. 2012;21(3):398-410.PubMedGoogle ScholarCrossref
38.
Usher-Smith  JA, Walter  FM, Emery  JD, Win  AK, Griffin  SJ.  Risk prediction models for colorectal cancer: a systematic review.  Cancer Prev Res (Phila). 2016;9(1):13-26.http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids=24895414&dopt=AbstractPubMedGoogle ScholarCrossref
39.
Spitz  MR, Etzel  CJ, Dong  Q,  et al.  An expanded risk prediction model for lung cancer.  Cancer Prev Res (Phila). 2008;1(4):250-254.PubMedGoogle ScholarCrossref
40.
Lughezzani  G, Briganti  A, Karakiewicz  PI,  et al.  Predictive and prognostic models in radical prostatectomy candidates: a critical analysis of the literature.  Eur Urol. 2010;58(5):687-700.PubMedGoogle ScholarCrossref
41.
Siontis  GC, Tzoulaki  I, Castaldi  PJ, Ioannidis  JP.  External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination.  J Clin Epidemiol. 2015;68(1):25-34.PubMedGoogle ScholarCrossref
42.
Kramer  AA, Zimmerman  JE.  Assessing the calibration of mortality benchmarks in critical care: the Hosmer-Lemeshow test revisited.  Crit Care Med. 2007;35(9):2052-2056.PubMedGoogle ScholarCrossref
43.
Davies  JR, Chang  YM, Bishop  DT,  et al.  Development and validation of a melanoma risk score based on pooled data from 16 case-control studies.  Cancer Epidemiol Biomarkers Prev. 2015;24(5):817-824.PubMedGoogle ScholarCrossref
44.
Collins  GS, Altman  DG.  Identifying patients with undetected colorectal cancer: an independent validation of QCancer (colorectal).  Br J Cancer. 2012;107(2):260-265.PubMedGoogle ScholarCrossref
45.
Li  K, Hüsing  A, Sookthai  D,  et al.  Selecting high-risk individuals for lung cancer screening—a prospective evaluation of existing risk models and eligibility criteria in the German EPIC cohort.  Cancer Prev Res (Phila). 2015;8(9):777-785.PubMedGoogle ScholarCrossref
46.
Foley  RW, Gorman  L, Sharifi  N,  et al.  Improving multivariable prostate cancer risk assessment using the Prostate Health Index.  BJU Int. 2016;117(3):409-417.PubMedGoogle ScholarCrossref
×