Dermoscopic images used for patient evaluation. A, Dermoscopic image of pigmented macule located on the leg. B, Machine-generated dermoscopic image of lesion in A. C, Histopathologic evaluation reveals a relatively broad asymmetric proliferation of mostly single and focally nested atypical melanocytes arranged along the dermoepidermal junction and at higher levels of the epidermis. A moderately dense lymphohistiocytic infiltrate is also evident in the subjacent dermis (hematoxylin-eosin, original magnification × 60). The histopathologic diagnosis was malignant melanoma in situ.
Diagnostic sensitivity and specificity of 10 dermoscopists on a set of 99 small pigmented skin lesions, including 49 melanomas. Error bars indicate 95% confidence intervals for each reader; the confidence intervals are smaller than interreader variability.
Biopsy sensitivity and specificity of 10 dermoscopists on a set of 99 small pigmented skin lesions, including 49 melanomas. Error bars indicate 95% confidence intervals for each reader; the confidence intervals are smaller than interreader variability. The result of the computer-vision system for the same set of lesions is also shown.
Friedman RJ, Gutkowicz-Krusin D, Farber MJ, Warycha M, Schneider-Kels L, Papastathis N, Mihm MC, Googe P, King R, Prieto VG, Kopf AW, Polsky D, Rabinovitz H, Oliviero M, Cognetta A, Rigel DS, Marghoob A, Rivers J, Johr R, Grant-Kels JM, Tsao H. The Diagnostic Performance of Expert Dermoscopists vs a Computer-Vision System on Small-Diameter Melanomas. Arch Dermatol. 2008;144(4):476-482. doi:10.1001/archderm.144.4.476
To evaluate the performance of dermoscopists in diagnosing small pigmented skin lesions (diameter ≤ 6 mm) compared with an automatic multispectral computer-vision system.
Blinded comparison study.
Dermatologic hospital-based clinics and private practice offices.
From a computerized skin imaging database of 990 small (≤ 6-mm) pigmented skin lesions, all 49 melanomas from 49 patients were included in this study. Fifty randomly selected nonmelanomas from 46 patients served as a control.
Main Outcome Measures
Ten dermoscopists independently examined dermoscopic images of 99 pigmented skin lesions and decided whether they identified the lesions as melanoma and whether they would recommend biopsy to rule out melanoma. Diagnostic and biopsy sensitivity and specificity were computed and then compared with the results of the computer-vision system.
Dermoscopists were able to correctly identify small melanomas with an average diagnostic sensitivity of 39% and a specificity of 82% and recommended small melanomas for biopsy with a sensitivity of 71% and specificity of 49%, with only fair interobserver agreement (κ = 0.31 for diagnosis and 0.34 for biopsy). In comparison, in recommending biopsy to rule out melanoma, the computer-vision system achieved 98% sensitivity and 44% specificity.
Differentiation of small melanomas from small benign pigmented lesions challenges even expert physicians. Computer-vision systems can facilitate early detection of small melanomas and may limit the number of biopsies to rule out melanoma performed on benign lesions.
Detection of early malignant melanoma (in situ and thin lesions) is one of the most effective ways of preventing mortality from this disease. Prognosis for patients with melanoma is dependent on early detection,1,2 as evidenced by the 10-year survival rates as high as 99.5% that have been reported for thin melanomas smaller than 0.76-mm thick in the New York University melanoma database; these rates markedly decrease to 48% for lesions larger than 3 mm in thickness.1 The effectiveness of this strategy is further confirmed because the reported marked reduction in mortality from melanoma, from 60% for those patients with melanoma diagnosed in 1960 to approximately 11% in 2005, is mainly due to early detection of thinner lesions followed by appropriate treatment.3,4
The incidence of melanoma in the general population is increasing in the United States and worldwide.3,4 Several reports5- 8 have also indicated the presence of small melanomas, defined as those with diameters of 6 mm or smaller. The mere presence of melanomas 6 mm or smaller warrants efforts to facilitate their identification because many of these small melanomas may appear benign by clinical criteria and are therefore more difficult to diagnose.9 In light of findings that smaller melanomas tend to be less deeply invasive than melanomas larger than 6 mm and, as such, generally have a more favorable prognosis,10,11 efforts that focus on early detection of small melanomas could potentially reduce mortality from this disease.
Supplementing clinical examination with techniques such as dermoscopy is an effective strategy for improving the sensitivity of dermoscopy-trained dermatologists in melanoma diagnosis.12 Although dermoscopy may be more effective than clinical examination in diagnosing thin malignant melanomas,13,14 a study by Carli et al15 demonstrated that dermoscopy did not improve diagnostic performance in identifying small (< 6-mm) melanomas. This finding indicates that additional diagnostic tools may be necessary to diagnose smaller melanomas more effectively.
Although many systems have been devised to facilitate the differentiation between histologically benign and malignant lesions, including dermoscopy and computer-assisted image analysis, this task continues to challenge even the most experienced dermatologist. The clinical ABCD criteria, initially described in 1985, remain an objective succinct algorithm for the early clinical recognition of melanoma.1 Recently, E for evolving was added to the ABCD criteria to highlight the element of change as an important diagnostic feature of cutaneous melanoma.11 The present study was designed to assess the sensitivity of dermoscopists in diagnosing small melanomas (≤ 6-mm diameter) compared with a novel automatic computer-vision system.
This study used cases from the digital dermoscopic database acquired by Electro-Optical Sciences Inc for the development and testing of MelaFind (Electro-Optical Sciences Inc, Irvington, New York), a computer-vision system for early detection of melanoma that is undergoing clinical testing. Twenty-six clinical sites in the United States and abroad have contributed to this database.
Only pigmented skin lesions (PSLs) were included in the database. The lesions were scanned using the multispectral computer-vision device before excisional or deep shave biopsy in toto. Approximately 80% of the lesions were biopsied to rule out melanoma, whereas the remaining lesions were biopsied mostly to rule out nonmelanoma skin cancer or because of patient concern. The diameter of eligible PSLs ranged from 2 to 22 mm. Previously biopsied, ulcerated, or bleeding lesions were excluded. Also excluded were lesions on mucosal surfaces and lesions that contained foreign matter (eg, tattoos).
Every case in the database consisted of multispectral dermoscopic images, a case record form with patient and lesion information (sex, age, and lesion location), prebiopsy diagnoses by the examining dermatologist, and a diagnostic histologic slide. Every slide was evaluated by 2 study dermatopathologists (from a panel of 4 dermatopathologists [M.C.M., P.G., R.K., or V.G.P.]) without knowledge of any additional clinical information; in cases of significant discordance in diagnoses, the slide was reviewed by a third study dermatopathologist. A lesion with at least 1 diagnosis of melanoma by the study dermatopathologists is considered melanoma. The histologic diagnoses distinguished invasive and in situ melanomas and high- and low-grade dysplastic nevi. Dysplastic nevi with severe cytologic atypia were considered high grade, and those with mild to moderate atypia were considered low grade.16 These diagnoses provided a reference standard by which the diagnostic performance of dermoscopists and of the computer-vision system was evaluated.
MelaFind is a multispectral digital dermoscope that, for every lesion, acquires 7 images in the visible spectral bands and 3 images in the near-infrared spectral bands. All images are analyzed automatically for the following: (1) calibration to determine the fraction of the incident radiation that is reflected for every pixel in the image; (2) image quality control that determines whether the images are suitable for further analysis (eg, a lesion covered with too much hair is automatically rejected and the operator is asked to clip the hair and retake the image); (3) segmentation to create a lesion mask; (4) computation of lesion properties in different spectral bands; and (5) lesion classification.17,18 The overall lesion classifier consists of 6 constrained linear classifiers, each trained to differentiate melanomas with 100% sensitivity from a particular type of lesion (low-grade dysplastic nevus, congenital nevus, common nevus, seborrheic keratosis, solar lentigo, and pigmented basal cell carcinoma). Thus, each lesion is characterized by 6 scores. A lesion is recommended for biopsy to rule out melanoma only if all scores are above the threshold value. On an independent testing set of 54 melanomas and 508 other PSLs not limited to small size, this classifier had biopsy sensitivity of 98% and specificity of 44%.
Small (≤ 6-mm) lesions were selected for the study in July 2005 from the database of 1977 eligible and evaluable PSLs, including 202 malignant melanomas. The lesion diameter was determined automatically by the computer-vision system.19 There were 990 (50% of the total) small lesions, of which 49 were melanomas; thus, 24% of all malignant melanomas were small. All PSLs were obtained either from the training database (75 small lesions, of which 38 were melanomas and 37 were matched nonmelanomas) or from the blinded set of data (24 small lesions, of which 11 were melanomas and 13 were nonmelanomas).
All 49 small malignant melanomas were included in this study. The small nonmelanomas were stratified by patient age (1-30 years, 31-60 years, and 60 years or older), sex (female or male), and lesion location (head and neck, trunk, lower limbs, or upper limbs); 50 nonmelanomas were selected randomly to match the frequency of these characteristics in the melanoma sample. High-grade dysplastic nevi were excluded because no consensus exists on their management. The distribution of lesions in the study is given in Table 1.
Participants in the study (readers) received a CD-ROM with color dermoscopic images created using MelaFind multispectral images. A ruler was included in every image to allow readers to determine the lesion diameter independently. For some of the cases, standard dermoscopic images acquired with a Nikon Coolpix 4300 camera (Nikon) with a 3Gen dermoscopic attachment (3Gen.LLC) were also available. The equivalence of standard and computer-vision system dermoscopic images was visually assessed by 3 readers (A.W.K., H.R., and A.C.) on a set of 10 lesions (2 melanomas and 8 low-grade dysplastic nevi). Overall, the consensus of the readers was that the computer-vision system images are suitable for dermoscopic evaluation (Figure 1).
For every lesion, information was provided about patient sex, age, and lesion location, whereas histologic information was not provided. The readers also received a form for recording their assessments of every lesion. All 10 readers were expert dermoscopists (9 dermatologists and 1 nurse practitioner specializing in dermatology), and their evaluations were performed independently.
To determine diagnostic performance, each reader had to answer the following question: “Is this lesion a melanoma?” Individual responses were then compared with the histopathologic diagnosis, which served as the reference standard for the determination of the diagnostic sensitivity and specificity for each reader.
Diagnostic performance does not provide information about case management by dermatologists. To obtain such information, the readers were asked to answer the following question: “Would you biopsy/excise this lesion?” If the answer was yes, the readers had to specify the reason for biopsy. As with evaluation of diagnostic performance, histologic diagnoses were used as the reference standard to evaluate lesion management decisions. If readers indicated that they would biopsy the lesion because they were sure it was melanoma or to rule out melanoma, then the case was considered true positive (TP) if the histologic diagnosis was melanoma and false positive (FP) otherwise. If the reader would not have biopsied the lesion or would have biopsied the lesion to rule out nonmelanoma skin cancer, the case was considered true negative (TN) if the histologic diagnosis was not melanoma and false negative (FN) otherwise. These data allow determination of biopsy sensitivity and specificity of the readers.
The diagnostic performance and lesion management decisions were analyzed by computing, for every reader, sensitivity (TP/[TP + FN]) and specificity (TN/[TN + FP]), as well as 95% confidence intervals, for these quantities. The interobserver variability was assessed using the κ statistic.20 The average diagnostic sensitivity and specificity and average biopsy sensitivity and specificity were also computed, with 95% confidence intervals determined according to the Obuchowski21 method, which takes into account correlations among the readers. For the averages, the other metrics of interest were positive predictive value (TP/[TP + FP]), negative predictive value (TN/[TN + FN]), and diagnostic accuracy (TP/[TP + FN + FP]). These variables were compared with the results of the automatic computer-vision system on the same set of small lesions.
Diagnostic sensitivity and specificity for all 10 readers (as well as averages) are displayed in Figure 2 in the format of a receiver operating characteristic plot. The error bars in Figure 2 represent 95% confidence intervals, which were large because of the relatively small sample size for each reader. Despite these large confidence intervals, the interreader variability was even larger; the κ statistic was 0.31, indicating only fair agreement among the readers. On small lesions, the average sensitivity to malignant melanoma was only approximately 40%, but the associated specificity was high (approximately 80%). The median diagnostic sensitivity and specificity were similar, at 43% and 84%, respectively.
The fact that diagnostic sensitivity to melanoma was only approximately 40% does not imply that dermatologists do not treat approximately 60% of small melanomas; it only means that this is not a good measure of lesion management decisions. Information about such decisions can be gained from biopsy sensitivity and specificity (Figure 3). The interreader variability on lesion management decisions was high: biopsy sensitivity ranged from 37% to 88%, whereas specificity varied from 22% to 80% (κ = 0.34). However, for each reader, biopsy sensitivity was higher than diagnostic sensitivity. This is not surprising because some dermoscopically borderline lesions may be called benign but are nevertheless biopsied to rule out melanoma. Because biopsy sensitivity is higher, it follows that biopsy specificity is lower.
The MelaFind database was randomly divided into the training and blind testing data sets. The biopsy sensitivity and specificity of MelaFind on the small lesions in the training set (38 melanomas and 37 nonmelanomas) was 100% and 46%, respectively. The biopsy sensitivity and specificity of the expert dermoscopist readers on average was 71% and 52%, respectively. On the small lesions in the blind testing set, the biopsy sensitivity of MelaFind was 91% (missed 1 melanoma in situ), with a specificity of 38%.
To increase the sample size of the small melanomas for a more robust statistical comparison of expert readers and computer-vision system, the small lesions from the MelaFind training and blinded data sets were combined. This pooling of data for MelaFind would be justified only if its results (sensitivity and specificity) are homogeneous for the 2 sets. Because of high sensitivity, the homogeneity assumption was tested using the Fisher exact test.22 Based on the values of P = .22 for MelaFind sensitivity and P = .75 for MelaFind specificity, the null hypothesis of homogeneity is valid and data were pooled. On average, the expert dermoscopist readers had a biopsy sensitivity of 71%, with a specificity of 49%. The median biopsy sensitivity and specificity of the 10 dermoscopists was 74% and 50%, respectively. The biopsy sensitivity and specificity of MelaFind was 98% (missed 1 melanoma in situ) and 44%, respectively.
Detailed comparison of human and computer vision for the management of small PSLs (biopsy sensitivity and specificity) is presented in Table 2. It clearly demonstrates that for small lesions, the computer-vision system had significantly higher sensitivity than dermoscopists (P < .001), while the difference in specificities was not statistically significant (P = .75). In addition, the computer-vision system had statistically significant (P = .02) higher values of negative predictive value; the differences in positive predictive value (P = .48) and diagnostic accuracy (P = .08) were not statistically significant.
Sensitivity to invasive and in situ melanomas should also be considered separately; the results are given in Table 3. The 95% confidence intervals were large because of the small sample sizes. Nevertheless, it was clear that dermoscopists and the computer-vision system have higher sensitivity to invasive than to in situ melanomas. The data in Table 3 indicate that approximately 19% of small invasive melanomas and approximately 37% of small melanomas in situ may be left unbiopsied, even by expert physicians.
Reports vary in their assessment of the prevalence of melanomas 6 mm or smaller among the overall population of cutaneous melanomas.5,7,8,10 In a retrospective review of small-diameter melanomas, Abbasi et al11 concluded that the prevalence of small melanomas (≤ 6 mm) ranged from less than 5% to 14%. The prevalence of small-diameter melanomas in our database, however, was much higher, at 25%, prompting us to evaluate this subset of lesions more closely. Although techniques of lesion recognition, such as dermoscopy, are gaining in popularity, the differentiation of early melanomas from benign PSLs continues to be plagued by uncertainty. This is especially true for the subset of small-diameter melanomas, which frequently display clinical and histologic discord.23 The present study is one of the first, to our knowledge, to quantify diagnostic accuracy of dermoscopists in specifically identifying small melanomas. By comparing the diagnostic performance of dermoscopists with a computer-vision system, the design of this study gives insight into new methods of identifying small melanomas in early stages of development.
In our study, we determined diagnostic sensitivity and specificity and biopsy sensitivity and specificity. This approach uniquely allowed us to differentiate between diagnostic performance and lesion management decisions. Although the average diagnostic sensitivity for all 10 dermoscopists was only 39%, the average biopsy sensitivity was 71%. A similar disparity was seen between the average diagnostic and biopsy specificities of 82% and 49%, respectively, reflecting the fact that lesions suggestive of disease are often biopsied to rule out melanoma. Despite the high biopsy sensitivity of our readers, nearly 30% of melanomas smaller than 6 mm would not have been biopsied. Furthermore, only fair interobserver agreement (κ = 0.31 for diagnosis and 0.34 for biopsy decision) indicates that dermoscopists differ in their evaluation of small PSLs, emphasizing the challenge in small melanoma diagnosis.
Only pigmented lesions scheduled for biopsy were eligible for inclusion in the database. Therefore, in the present study, the true biopsy sensitivity of examining dermatologists could not be determined directly. Thus, the pooled biopsy sensitivity was artificially high and specificity was relatively low. One could, in fact, take an extreme position that the examining physicians had 100% biopsy sensitivity and 0% biopsy specificity on the lesions in the database. The present study determined that the average biopsy sensitivity to small melanomas of the readers was 71%, and the corresponding specificity was 49%. However, only 2 small melanomas in situ were missed by all readers (ie, the combined biopsy sensitivity was 96%). At the same time, the combined biopsy specificity was 6% (only 3 lesions were TN for all readers). The combined biopsy sensitivity and specificity are, therefore, similar to the pooled high sensitivity and low specificity of more than 30 examining dermatologists, who contributed to the database using a variety of approaches to decide on lesion management. Thus, the sensitivity to melanoma can be increased by combining evaluations of lesions by multiple physicians but at the cost of reduced specificity. In practice, this means that, if a patient could consult 10 dermatologists about a single pigmented lesion, the probability that a small melanoma would be biopsied is 96%. However, if a patient is examined by a single dermatologist then the probability that a small melanoma would be biopsied is 71%.
Previously reported sensitivities and specificities for the clinical diagnosis of melanoma are higher than our findings, with sensitivities ranging anywhere from 58% to more than 90%12,14,24- 26 and specificities ranging from 77% to 99%.26,27 One must take into account, however, that most published studies on diagnostic performance are not limited to small-diameter PSLs. This could explain why our results decreased below those frequently encountered in the literature. In a study28 that evaluated only melanomas smaller than 7 mm in diameter, the diagnostic sensitivity was 44%, which is more consistent with our findings. Furthermore, variability in sensitivity measurements could reflect the different proportions of in situ and invasive lesions that are included in a given study on diagnostic performance. It is likely that dermatologists diagnose thicker more advanced lesions with greater sensitivity. In our study, 21 of the 49 small melanomas were invasive, with thicknesses ranging from 0.1 to 1.4 mm and a median thickness of 0.32 mm. Similar findings were reported by Kamino et al,5 in which a sample of 30 small-diameter melanomas had thicknesses ranging from 0.25 to 1.4 mm. In our study, the diagnostic and biopsy sensitivity of expert dermoscopists in the evaluation of small-diameter invasive melanomas was 48% and 81%, respectively, significantly higher than the diagnostic and biopsy sensitivity for in situ melanomas, which was 33% and 63%, respectively.
A number of studies have also reported on the sensitivity of melanoma diagnosis when combining dermoscopy with naked-eye examination. In a study by Carli et al,15 in which melanocytic lesions were segregated according to lesion diameter, dermatologists identified melanomas smaller than 6 mm with a sensitivity of 64% when using clinical and dermoscopic examination. However,they found that the addition of dermoscopy to naked-eye examination did not provide a statistically significant improvement in sensitivity. In contrast, Bono et al8 reported that the addition of dermoscopy to the clinical assessment of small-diameter melanomas allowed for a higher rate of recognition, from 49% with naked-eye examination to 72% with dermoscopic techniques. Others12,29 have also reported improvement in the diagnostic accuracy with the use of dermoscopy, especially in small lesions. The discordant findings among these studies could be attributable to the different levels of expertise of examiners, to different proportions of invasive and in situ melanomas, and to differences in the prevalence of melanoma included in the sample. In a meta-analysis that evaluated the dermoscopic assessment of pigmented lesions, Kittler et al14 surmised that, as the proportion of melanoma cases that composed a study subset increased, the diagnostic difficulty of the sample increased.
Our results must be examined within the context of the clinical presentation of small melanomas, which often make them more difficult to identify than larger-diameter malignant melanomas. Thomas et al30 found that the specificity of melanoma diagnosis increases with the number of ABCDE criteria present. Small melanomas, however, do not always display the features typically used to diagnose malignant melanoma. Bergman et al10 found that small melanomas have a different clinical presentation than large melanomas because smaller lesions do not always exhibit the characteristic ABCD features, and others31,32 suggested that malignant melanomas do not display typical clinical and histologic features of melanoma until the lesions are larger than 5 mm. Because morphologic features that help distinguish melanomas from benign lesions may not be visible to the naked eye,29 it is not surprising that, in light of their clinical presentation, small melanomas are missed much more frequently. The present study found that small melanomas are also difficult to diagnose by dermoscopic evaluation. Given these circumstances, emphasis should, thus, be placed on the E criterion when evaluating small pigmented lesions, which Abbasi et al11 defined as evolving (ie, change over time in size, shape, color, surface features, or symptoms). In fact, Kamino et al5 reported that the most frequent reason for excision of small melanomas was a new or changing lesion. Likewise, Helsing and Loeb28 noted that a change in color was more frequently seen in small melanomas than in larger melanomas.
In conclusion, the differentiation of small melanomas from small benign pigmented lesions challenges even expert physicians. In comparing human evaluation with a computer-vision system, we found that the computer-vision system recommended biopsy for 98% of small melanomas (1 melanoma in situ missed), whereas dermoscopists, on average, would biopsy only 71% of small melanomas (29% missed, both in situ and invasive). By analyzing features indiscernible to the human eye, automated systems could assist dermatologists in the selection of lesions suggestive of disease for biopsy to rule out melanoma. Not only will the addition of such diagnostic tools limit the number of biopsies necessary to rule out melanoma on clinically suspicious yet histologically benign pigmented lesions, it will also aid in the detection and treatment of melanomas during the curable stages of development.
Correspondence: Robert J. Friedman, MD, Department of Dermatology, New York University School of Medicine, 124 E 72nd St, New York, NY 10021.
Accepted for Publication: December 7, 2007.
Author Contributions: Dr Friedman had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Friedman, Gutkowicz-Krusin, Farber, Warycha, Schneider-Kels, and Prieto. Acquisition of data: Gutkowicz-Krusin, Mihm, Googe, King, Prieto, Polsky, Rabinovitz, Oliviero, Cognetta, Rivers, and Tsao. Analysis and interpretation of data: Friedman, Gutkowicz-Krusin, Farber, Warycha, Schneider-Kels, Papastathis, Mihm, Googe, King, Prieto, Kopf, Rabinovitz, Oliviero, Rigel, Marghoob, Johr, and Grant-Kels. Drafting of the manuscript: Friedman, Gutkowicz-Krusin, Farber, Warycha, Schneider-Kels, Papastathis, and Prieto. Critical revision of the manuscript for important intellectual content: Friedman, Gutkowicz-Krusin, Farber, Warycha, Schneider-Kels, Papastathis, Mihm, Googe, King, Prieto, Kopf, Polsky, Rabinovitz, Oliviero, Cognetta, Rigel, Marghoob, Rivers, Johr, Grant-Kels, and Tsao. Statistical analysis: Gutkowicz-Krusin. Obtained funding: Friedman. Administrative, technical, and material support: Gutkowicz-Krusin, Farber, Warycha, Schneider-Kels, Papastathis, and Cognetta. Study supervision: Friedman, Kopf, Rigel, Grant-Kels, and Tsao.
Financial Disclosure: Dr Friedman is a shareholder, former director, consultant, and member of the Scientific Advisory Committee for Electro-Optical Sciences Inc (EOS); Dr Gutkowicz-Krusin is an employee of EOS; Mss Farber and Schneider-Kels and Dr Warycha received research support from EOS; Dr Mihm is a member of the Scientific Advisory Committee and the chief dermatopathologist for EOS; Drs Googe and King are dermatopathologists for EOS studies; Dr Prieto is a dermatopathologist for and is involved in a research project with EOS; Dr Kopf is a former member of the Scientific Advisory Committee for EOS; Drs Polsky and Rabinovitz are consultants, principal investigators, and members of the Scientific Advisory Committee for EOS; Dr Cognetta is a principal investigator and member of the Scientific Advisory Committee for EOS; Dr Rigel is a consultant and member of the Scientific Advisory Committee for EOS; and Dr Grant-Kels will be a future investigator for EOS.
Funding/Support: Mss Farber and Schneider-Kels and Dr Warycha received research support from EOS.
Role of the Sponsor: Electro-Optical Sciences Inc provided data to the research team but had no role in study design, data analysis, data interpretation, or writing of the report.
Additional Contributions: Alicia Toledano, ScD, helped with statistical analyses and Nikolai Kabelev, BCSc, Joanna Adrian, MBA, and Mrinalini Roy, BA, of Electro-Optical Sciences Inc helped prepare the data for this study.