Design of the study and patients' enrollment. Asterisk indicates referred sample; dagger, random sample.
Flowchart describing the management options after the evaluation of teleconsultations. DCL indicates diagnostic confidence level.
Customize your JAMA Network experience by selecting one or more topics from the list below.
Moreno-Ramirez D, Ferrandiz L, Nieto-Garcia A, et al. Store-and-Forward Teledermatology in Skin Cancer Triage: Experience and Evaluation of 2009 Teleconsultations. Arch Dermatol. 2007;143(4):479–483. doi:10.1001/archderm.143.4.479
To evaluate a store-and-forward teledermatology system aimed at the routine triage of patients with skin cancer.
A multicenter, longitudinal, 4-phase, descriptive and evaluation study of a referred sample of patients attended through teleconsultation between March 2004 and July 2005 (n = 2009). Clinical and dermoscopical examination and histopathological study were considered the gold standard.
A skin cancer unit of a public university hospital and 12 primary care centers in southern Spain.
The study population comprised patients with circumscribed lesions fulfilling at least 1 of the following criteria: changes in ABCD criteria (asymmetry, border irregularity, color variegation, and diameter >6 mm), recent history, multiple melanocytic lesions, symptoms, and/or patient's application for surgical treatment and concern about moles.
Diagnosis, diagnostic category (malignant lesions, high-risk lesions, benign lesions, special lesions, and other lesions), diagnostic confidence level on a 3-point scale, and management decision (referral vs nonreferral) were listed after the evaluation of each teleconsultation. A face-to-face evaluation and biopsy of selected patients were performed.
Main Outcome Measures
The filtering percentage, as the percentage of patients not referred to the face-to-face clinic, as well as waiting intervals and pick-up or skin cancer detection rates were evaluated as effectiveness indicators. Reliability measures (κ agreement), accuracy, and diagnostic performance indicators (validity) were also evaluated.
The filtering percentage was 51.20% (95% confidence interval [CI], 49.00%-53.40%). The waiting interval to attend the clinic was 12.31 days (95% CI, 8.22-16.40 days) through teledermatology and 88.62 days (95% CI, 38.42-138.82 days; P<.001) for the letter referral system. Pick-up rates were 2.02% (95% CI, 1.10%-2.94%) for malignant melanoma and 27.94% (95% CI, 24.98%-30.90%) or 1:3.71 for patients with any malignant or premalignant lesion. Intraobserver agreement was κ = 0.91 (95% CI, 0.89-0.93) for the management decision and κ = 0.95 (95% CI, 0.94-0.96) for the diagnosis. Interobserver concordance was κ = 0.83 (95% CI, 0.78-0.88) for the management decision and κ = 0.85 (95% CI, 0.79-0.91) for the diagnosis. Accuracy was κ = 0.81 (95% CI, 0.78-0.84). Sensitivity was 0.99 (95% CI, 0.98-1.00); specificity, 0.62 (95% CI, 0.56-0.69); pretest likelihood, 0.42 (95% CI, 0.37-0.47); positive posttest likelihood, 0.65 (95% CI, 0.61-0.69); and negative posttest likelihood, 0.01 (95% CI, 0.00-0.05).
Store-and-forward teledermatology has demonstrated in this series to be an effective, accurate, reliable, and valid approach for the routine management of patient referrals in skin cancer and pigmented lesion clinics.
For the last 10 years, numerous studies have presented teledermatology (TD) as a helpful complementary tool in different clinical settings. From general real-time TD consultation to more specific store-and-forward triage systems, all have been experienced with variable benefits improving patient's standards of care and, in terms of effectiveness, reliability, accuracy, and patients' satisfaction.1-3
A specific application of TD that has been repeatedly tested is the use of store-and-forward systems aimed at the management of patient referrals in skin cancer clinics. The diagnostic advantage of skin growths against generalized dermatoses in TD has been argued in favor of the implementation of such dermatologist-directed triage systems.1,4,5 However, as it has been recently reported, official recommendations promulgated for the early diagnosis of cancer (ie, the British “2-week rule”)6,7 are not working well, which is probably related to the lack of an appropriate referral system.8 In regard to this point, TD would allow the dermatologist to make decisions based on clinical information and Internet-transmitted digital pictures.
A store-and-forward teledermatology (SFTD) triage system aimed at the selection of patients with skin growths suggestive of cancer was implemented at our skin cancer clinic in 2003; the facility currently covers a total population of 300 000 inhabitants of a southern Spanish province living 5 to 100 km away, and it has turned into an essential complementary tool for our daily practice at the clinic.2
The present study describes the results of the implemented SFTD triage system in terms of clinical effectiveness, validity, and accuracy.
A multicenter, longitudinal, descriptive, and evaluation study of an SFTD system aimed at the triage of patients with skin cancer was conducted at the Pigmented Lesion and Skin Cancer Clinic of the University Hospital Virgen Macarena, Seville, Spain, and 12 primary care centers (PCCs) between March 2004 and July 2005. Clinical effectiveness, reliability, accuracy, and validity of SFTD as a triage and diagnostic tool were evaluated following the design and phases depicted in Figure 1.
During the first phase of the study, the first dermatologist (D.M.-R.) carried out the evaluation of all the teleconsultations received during the study period (n = 2009). The clinical diagnosis, diagnostic category, diagnostic confidence level, the management decision, and the quality of the pictures and clinical information submitted were all recorded after this first evaluation. The same items were recorded in a second evaluation of these teleconsultations by the same dermatologist to assess the intraobserver concordance. In this phase, teleconsultations received within the last 3 months of the study (March-July 2005, inclusive) were excluded to impose a 3- to 12-month period between the 2 evaluations to lower the possibility of intraobserver bias (n = 1589). Teleconsultations were also randomly shuffled prior to the second evaluation by the same dermatologist.
During the third phase of the study, 2 samples of patients were evaluated at the face-to-face clinic (Figure 1). A random sample of patients (n = 403) attended through teleconsultations was referred to the face-to-face clinic regardless of the benign or malignant diagnosis reached. The second sample of patients consisted of all the patients routinely referred and seen at the face-to-face clinic (n = 882), from which the accuracy was calculated as the concordance between the diagnoses given after the first evaluation of teleconsultations and the gold standard. In a fourth phase, the second dermatologist (L.F.) evaluated a random sample of teleconsultations (n = 340) to complete the reliability study through the assessment of the diagnostic and management interobserver concordance.
Patients included in the study had to present at the participant PCC with circumscribed lesions fulfilling at least 1 of the following criteria: changes in ABCD criteria (asymmetry, border irregularity, color variegation, and diameter >6 mm) or growing lesion, recent history (<3 years), multiple lesions (>20 melanocytic nevi counted by the general practitioner),9 symptoms (eg, pain, itching, and bleeding), and patient request for surgical treatment and concern about moles.
According to established guidelines, after obtaining the consent form, 2 digital pictures, a panoramic view and a macrophotograph, are taken of these patients (Nikon Coolpix 4300 [1600 × 1200 pixels]; Nikon Corp, Tokyo, Japan).10 Pictures are inserted in a Microsoft Word (Microsoft Inc, Redmond, Wash) document, where clinical information is also gathered. This document is then sent via the Intranet ATM (Asynchronous Transfer Mode), ISDR-B (Integrated Services Digital Network type B), and Frame Relay/ADSL (Asymmetric Digital Subscriber Line) networks to the e-mail account of the clinic. After the evaluation of the teleconsultation using a 19-in (48.3-cm) thin-film transistor (TFT) monitor, a report with the possible diagnosis and management of the case is returned.
Diagnostic categories and diagnoses considered included the following:
Malignant and premalignant lesions (malignant melanoma including melanoma in situ and lentigo maligna, basal cell carcinoma, squamous cell carcinoma, keratoacanthoma, actinic keratosis and cheilitis, and other tumors)
Suspicious or high-risk lesions or phenotypes (clinically atypical nevus and multiple melanocytic nevi)
Benign lesions and melanocytic and nonmelanocytic clinically typical circumscribed lesions (common acquired melanocytic nevus, seborrheic keratosis, dermatofibroma, blue nevus, actinic lentigo, vascular lesions, and others)
“Special” lesions and melanocytic and nonmelanocytic circumscribed lesions of uncertain potential (congenital nevus, acral nevus, spilus nevus, Sutton nevus, persistent nevus, mucosal nevus and lentigo, and others)
Other lesions (lesions and conditions not placed in previous categories)
The diagnostic confidence level was rated on a 3-point scale, with 3 indicating an absolutely confident clinical diagnosis and 1, an absolutely uncertain diagnosis. Management options were limited to the “referral” vs “nonreferral” of patients to the face-to-face clinic as shown in Figure 2. The quality of the pictures and the clinical information transmitted was graded as excellent, sufficient, and insufficient for the decision-making process. All the patients presenting with benign lesions after the in vivo evaluation who were referred to the face-to-face clinic and who did not warrant further intervention or follow-up were considered under the concept of unnecessary referral.
The time interval to attend the face-to-face clinic since the first visit to the general practitioner was calculated. This period was compared with the mean waiting interval of a random sample of patients attended through the conventional letter referral system and fulfilling the aforementioned inclusion criteria (n = 530). The time the dermatologist spent reporting teleconsultations was also considered. The filtering percentage was calculated as the percentage of patients not referred to the face-to-face clinic out of the total number of teleconsultations received. The pick-up or detection rates of melanoma, nonmelanoma skin cancer, and category 1 lesions were calculated as the number of these types of lesions diagnosed among the patients seen at the face-to-face clinic after the triage performed by TD. The 2 × 2 contingency table constructed for the validity study and the diagnostic performance indicators calculated are given in Table 1 and Table 2, respectively.
A double gold standard was defined in this study. In those patients with clinically and dermoscopically benign and nonsuspicious lesions, with a diagnostic confidence level of 3 after the in vivo evaluation carried out by the 3 participant dermatologists (D.M.-R., L.F. and F.C.M.), this clinical and dermoscopical diagnosis was considered the gold standard. In cases of a diagnostic confidence level lower than 3 at the face-to-face examination, as well as in discordant cases after the in vivo evaluation by the participant dermatologists and in any type of malignant lesion or lesion suggestive of cancer, the gold standard was the histopathological study.
For statistical analysis, SPSS 12.0 software (SPSS Inc, Chicago, Ill) was used. The aforementioned concordances were calculated as Cohen κ values with 95% confidence intervals (CIs).11 Statistically significant differences were demonstrated by means of the χ2 test, and the t test for paired data was used to compare the diagnostic confidence level. The normal distribution of the data was analyzed by the Kolmogorov-Smirnov normality test. All significance tests were 2-sided, with P<.05 considered statistically significant. The Standards for Reporting of Diagnostic Accuracy guidelines were followed for the validity study.12
Over a period of 15 months, 2009 patients were attended through teleconsultation at the clinic, with a mean of 133.9 teleconsultations per month (range, 40-262; 95% CI, 106.95-160.91). Age and sex distribution of patients and the reason for teleconsultation are given in Table 3 and Table 4. Diagnostic categories and diagnoses found at the PCC, teleconsultation, and face-to-face clinic are given in Table 5.
Teleconsultation demonstrated a filtering percentage of 51.20% (95% CI, 49.00%-53.40%), with 980 patients (48.80%) being referred to the face-to-face clinic. Of these patients, 9.18% did not keep their appointments for several reasons, leading to a final sample of patients seen at the face-to-face clinic of 890 patients. Teleconsultations evaluated by the first dermatologist demonstrated a mean diagnostic confidence level of 2.59 (95% CI, 1.98-3.00) after the first evaluation and 2.62 (95% CI, 2.05-3.00) after the second evaluation (P<.001). The diagnostic confidence level for category 1 lesions was of 2.51 (95% CI, 1.93-3.00) after the first evaluation through TD. The diagnostic confidence level after the in vivo evaluation at the face-to-face clinic increased to 2.84 (95% CI, 2.45-3.00) (P<.001).
Teleconsultation reports were available to the general practitioner in a mean time interval of 61.06 hours (95% CI, 33.83-88.29 hours; range, 6-144 hours), with a 95th percentile of 96.00 hours. Patients referred to the face-to-face clinic were attended within a mean period of 12.31 days (95% CI, 8.22-16.40 days; range, 2-31 days), with a 95th percentile of 21.00 days. The mean interval for the sample of patients referred through the conventional letter referral system (n = 530) was 88.62 days (95% CI, 38.42-138.82 days; P<.001), with a 75th percentile of 120.00 days.
The dermatologist responsible for the first evaluation of teleconsultations spent a mean time of 1 hour to deal with 19.8 teleconsultations (mean of 3.03 minutes per teleconsultation; 95% CI, 1.23-4.83 minutes). The melanoma pick-up rate at the face-to-face clinic was 2.02% (95% CI, 1.10%-2.94%) or 1:49.89 patients seen at the clinic. The nonmelanoma skin cancer pick-up rate was 13.56% (95% CI, 11.30%-15.82%) or 1:7.36 patients seen at the clinic, and the premalignant and malignant lesions (category 1) pick-up rate was 27.94% (95% CI, 24.98-30.90) or 1:3.71 patients seen at the face-to-face clinic. Of the patients referred to the clinic, 71.21% presented cutaneous lesions that warranted surgical or medical therapy or periodic follow-up at the clinic. The remaining 28.79% of patients did not require any intervention and were discharged from the clinic (unnecessary referrals).
The quality of the digital pictures transmitted was excellent in up to 21.50% (n = 432) of the teleconsultations (95% CI, 19.70%-23.30%). In 73.00% (n = 1466) of cases (95% CI, 71.06%-74.94%), the transmitted pictures were appropriate, with 5.55% (95% CI, 4.55%-6.55%; n = 111) of pictures not of sufficient quality for the decision-making process. Of the teleconsultations, the clinical information transmitted was excellent in 6.60% (n = 133), appropriate in 90.80% (n = 1824), and poor in 2.60% (n = 52). The lack of information regarding the time evolution of the lesions accounted for 72.50% (n = 38) of the teleconsultations with poor clinical information.
The reliability study yielded the following results: intraobserver agreement was κ = 0.91 (95% CI, 0.89-0.93) for the management decision and κ = 0.95 (95% CI, 0.94-0.96) for the diagnoses given by the first dermatologist. Interobserver concordance was κ = 0.83 (95% CI, 0.78-0.88) when the management options were assessed and κ = 0.85 (95% CI, 0.79-0.91) for the diagnosis. The second dermatologist demonstrated a lower diagnostic confidence level (2.59 vs 2.66; P<.001) and a lower filtering percentage (50.60% vs 51.20%; P<.05).
Agreement between the teleconsultation diagnosis and the gold standard was κ = 0.81 (95% CI, 0.78-0.84). This diagnostic accuracy increased to 0.89 (95% CI, 0.84-0.94) for category 1 lesions. The agreement between the diagnosis made by the general practitioner and that of the dermatologist by teleconsultation was κ = 0.46 (95% CI, 0.43-0.49). Validity measures obtained in this study are given in Table 2.
Although the application of TD as a triage system for skin cancer is not new, long-term studies of TD working as a routine tool for the daily practice of skin cancer clinics at public hospitals are lacking. This use of TD therefore warrants a thorough evaluation in terms of clinical effectiveness, reliability, and validity.
Regarding the effectiveness of TD systems, the lack of quantifiable clinical end points that may be applied to different clinical situations (eg, mortality and quality of life) has turned intermediate clinical outcomes (eg, consultations avoided, time to intervention, and consultation time requirements) into the most descriptive indicators of the clinical effectiveness of TD systems.13
Mean waiting intervals reported in studies evaluating TD systems have ranged between 2 and 50 days for TD systems vs the 88 to 137 days demonstrated by the conventional letter referral.2,14-16 In our series, patients referred to the clinic, one third of whom had malignant lesions, were attended to within the following 2 weeks (mean time, 12.31 days) since they first visited the general practitioner, in accordance with the 2-week rule promoted by health care administrations.6,7 Along with the faster communication channel that the Internet represents, the avoidance of unnecessary visits to the dermatologist may explain such shortening of waiting intervals.
In previous studies, clinic-based evaluations avoided ranged between 44% and 82% for real-time TD systems and between 18% and 31% for the store-and-forward method, with the greater amount of clinical information provided by videoconference being suggested as the main reason for this advantage in referrals avoided.1,14,17-19 In our study, although it involved a store-and-forward facility, 94.45% of teleconsultations retained enough information to make a decision, leading to a 51.20% of visits avoided, which is closer to the rates achieved by real-time facilities compared with previously published SFTD experiences.14 The triage objective of the evaluated facility, along with the standard referral forms developed to transmit clinical information, may account for this advantage.
The malignant melanoma pick-up rate disclosed in the present study agrees with previous studies reporting rates between 1:22 and 1:64 in European pigmented lesion clinics and largely improves the detection rate reported in US clinics (1:250).20,21 However, to our knowledge, no previous studies provide information about the nonmelanoma skin cancer pick-up rate. In that respect, after the TD-based triage, 1 of 3.7 patients seen at the clinic presented with any type of premalignant or already malignant lesion that warranted intervention.
Regarding the reliability assessment, high simple agreement rates ranging from 55% to 100% have been reported for biopsy decisions in SFTD-based skin cancer triage systems.3,22-25 According to κ statistics, the strength of the diagnostic agreement has also ranged from moderate to perfect in similar previous experiences.2,3
The evaluated triage system has also disclosed excellent agreement in terms of concordance κ values between face-to-face and Internet-based diagnosis (κ = 0.81), a concept that is also called accuracy in other series. Available studies have also calculated concordances ranging from 81% to 89% between face-to-face dermatologists and store-and-forward dermatologists,13,26 with other series reporting similar degrees of agreement between TD and the traditional consultation.13,27 It is worthwhile to mention a recent study that reported a 48% agreement between TD and conventional consultation.22 We suggest that the notable proportion of poor images and the lack of lesion details given in the referral teleconsultation template were probably related to the limited diagnostic accuracy of SFTD for skin lesions in these authors' findings.22
Few studies on TD have dealt with the evaluation of validity in terms of sensitivity and specificity.3,23,28 Moreover, to our knowledge, no evaluation study of TD has dealt with other indicators of diagnostic performance recommended by the STARD (Standards for Reporting of Diagnostic Accuracy) guidelines, which may strengthen the results obtained by the simple assessment of sensitivity and specificity.12 In these terms, the finding of a posttest likelihood higher than the pretest likelihood suggests that this tool is useful for the diagnosis of skin cancer. In addition, the low negative posttest likelihood obtained also indicates the low likelihood of a nonreferred patient’s having a skin cancer.
In conclusion, the facility evaluated in this series involves an SFTD triage system aimed at the routine case management of patients presenting with cutaneous growths suggestive of cancer at their visit to their general practitioner. After the evaluation of 2009 teleconsultations, SFTD has resulted in an effective, reliable, and valid triage tool, suitable to be integrated into the routine practice of skin cancer clinics.
Correspondence: David Moreno-Ramirez, PhD, Pigmented Lesion and Teledermatology Clinic, Dermatology Department, University Hospital Virgen Macarena, Avda Dr Fedriani s/n, 41009 Seville, Spain (email@example.com).
Financial Disclosure: None reported.
Accepted for Publication: September 4, 2006.
Author Contributions: Drs Moreno-Ramirez and Ferrandiz had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Moreno-Ramirez, Ferrandiz, Carrasco, Moreno-Alvarez, Galdeano, and Camacho. Acquisition of data: Moreno-Ramirez, Ferrandiz, Moreno-Alvarez, Galdeano, and Rios-Martin. Analysis and interpretation of data: Moreno-Ramirez, Ferrandiz, Galdeano, and Camacho. Drafting of the manuscript: Moreno-Ramirez and Ferrandiz. Critical revision of the manuscript for important intellectual content: Moreno-Ramirez, Ferrandiz, Carrasco, Moreno-Alvarez, Galdeano, Rios-Martin, and Camacho. Statistical analysis: Moreno-Ramirez and Ferrandiz. Obtained funding: Moreno-Ramirez, Carrasco, and Camacho. Administrative, technical, and material support: Moreno-Ramirez, Carrasco, Moreno-Alvarez, Galdeano, and Camacho. Study supervision: Moreno-Ramirez, Rios-Martin, and Camacho.
Funding/Support: This study was supported by grant FIS PI 04/1194 from the Instituto Carlos III.
Create a personal account or sign in to: