Original Investigation
December 2015

Detection Accuracy of Collective Intelligence Assessments for Skin Cancer Diagnosis

Author Affiliations
  • 1Department of Biology and Ecology of Fishes, Leibniz-Institute of Freshwater Ecology and Inland Fisheries, Berlin, Germany
  • 2Center for Adaptive Rationality, Max Planck Institute for Human Development, Berlin, Germany
  • 3Faculty of Life Sciences, Humboldt-University of Berlin, Berlin, Germany
  • 4Department of Dermatology, Second University of Naples, Naples, Italy
  • 5Department of Dermatology and Venereology, Medical University of Graz, Graz, Austria

Copyright 2015 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.

JAMA Dermatol. 2015;151(12):1346-1353. doi:10.1001/jamadermatol.2015.3149

Importance  Incidence rates of skin cancer are increasing globally, and the correct classification of skin lesions (SLs) into benign and malignant tissue remains a continuous challenge. A collective intelligence approach to skin cancer detection may improve accuracy.

Objective  To evaluate the performance of 2 well-known collective intelligence rules (majority rule and quorum rule) that combine the independent conclusions of multiple decision makers into a single decision.

Design, Setting, and Participants  Evaluations were obtained from 2 large and independent data sets. The first data set consisted of 40 experienced dermoscopists, each of whom independently evaluated 108 images of SLs during the Consensus Net Meeting of 2000. The second data set consisted of 82 medical professionals with varying degrees of dermatology experience, each of whom evaluated a minimum of 110 SLs. All SLs were evaluated via the Internet. Image selection of SLs was based on high image quality and the presence of histopathologic information. Data were collected from July through October 2000 for study 1 and from February 2003 through January 2004 for study 2 and evaluated from January 5 through August 7, 2015.

Main Outcomes and Measures  For both collective intelligence rules, we determined the true-positive rate (ie, the hit rate or specificity) and the false-positive rate (ie, the false-alarm rate or 1 − sensitivity) and compared these rates with the performance of single decision makers. Furthermore, we evaluated the effect of group size on true- and false-positive rates.

Results  One hundred twenty-two medical professionals performed 16 029 evaluations. Use of either collective intelligence rule consistently outperformed single decision makers. The groups achieved an increased true-positive rate and a decreased false-positive rate. For example, individual decision makers in study 1, using the pattern analysis as diagnostic algorithm, achieved a true-positive rate of 0.83 and a false-positive rate of 0.17. Groups of 3 individuals achieved a true-positive rate of 0.91 and a false-positive rate of 0.14. These improvements increased with increasing group size.

Conclusions and Relevance  Collective intelligence might be a viable approach to increase diagnostic accuracy in skin cancer and reduce skin cancer–related mortality.