Figure. Cutaneous Sarcoidosis Activity and Morphology Index.
Rosenbach M, Yeung H, Chu EY, Kim EJ, Payne AS, Takeshita J, Vittorio CC, Wanat KA, Werth VP, Gelfand JM. Reliability and Convergent Validity of the Cutaneous Sarcoidosis Activity and Morphology Instrument for Assessing Cutaneous Sarcoidosis. JAMA Dermatol. 2013;149(5):550-556. doi:10.1001/jamadermatol.2013.60
Author Affiliations: Department of Dermatology (Drs Rosenbach, Chu, Kim, Payne, Takeshita, Vittorio, Wanat, Werth, and Gelfand and Mr Yeung) and Center for Clinical Epidemiology and Biostatistics (Dr Gelfand), University of Pennsylvania Perelman School of Medicine, Philadelphia.
Importance A validated scoring system is essential to assess the effect of therapeutic interventions on a disease. The instrument introduced here captures sarcoidosis disease activity in a reliable, reproducible manner, which will help standardize clinical trial outcomes and allow comparative efficacy studies in the future and may help lead to more robust data regarding the effect of different treatments on cutaneous sarcoidosis.
Objective To assess the reliability and convergent validity of the Cutaneous Sarcoidosis Activity and Morphology Instrument (CSAMI) and Sarcoidosis Activity and Severity Index (SASI) for evaluating cutaneous sarcoidosis outcomes.
Design and Setting Cross-sectional study evaluating cutaneous sarcoidosis disease severity using CSAMI, SASI, and Physician's Global Assessment (PGA) as reference in the dedicated cutaneous sarcoidosis clinic of a teaching hospital.
Participants Eight dermatologists evaluating cutaneous sarcoidosis in 11 patients.
Intervention Evaluation using the study instruments.
Main Outcomes and Measures Primary outcomes included interrater and intrarater reliability and convergent validity; secondary outcomes, correlation with quality-of-life measures and time required for completion.
Results All instruments demonstrated good to excellent intrarater reliability. Interrater reliability was excellent for CSAMI Activity scores (intraclass correlation coefficient, 0.82 [95% CI, 0.66-0.94]) and fair to poor for CSAMI Damage scores (0.42 [0.21-0.72]), modified Facial SASI (0.40 [0.17-0.72]), and PGA scores (0.40 [0.18-0.70]). CSAMI Activity and Damage scores and modified Facial SASI all demonstrated convergent validity with statistically significant correlations with PGA scores. Trends for correlations were seen between CSAMI scores and specific Skindex-29 quality-of-life domains. Although CSAMI required longer time to complete than SASI, both were scored within adequate time for use in clinical trials.
Conclusions and Relevance CSAMI appears to be a reliable and valid outcome instrument to measure cutaneous sarcoidosis and may capture a wide range of body surface and cutaneous morphologic types. This instrument can be adopted into clinical practice and clinical trials to allow physicians to assess the intensity of their patients' cutaneous sarcoidosis disease activity. Widespread use of one metric for disease severity assessment can help standardize the evaluation of the effect of various treatments on the disease. Future research is necessary to demonstrate its sensitivity to change and to confirm its correlation with quality-of-life measures.
Sarcoidosis is an uncommon multisystem disease of unknown etiology characterized by granulomatous infiltrates, with cutaneous involvement occurring in 25% to 30% of cases.1 Recent advances in the pathogenesis of cutaneous sarcoidosis have resulted in broadening therapeutic options; nevertheless, most treatment recommendations are based on anecdotal reports or small case series, with a general lack of high-quality evidence for clinical efficacy.2 The development and validation of standardized disease severity instruments are required for rigorous clinical research and evidence-based dermatology.3
To our knowledge, no criterion standard instrument has been established for the objective assessment of severity of cutaneous sarcoidosis. Many previous studies about sarcoidosis treatments relied on subjective assessments of photographs or clinical descriptors, making objective interpretation of response challenging.4- 7 The Sarcoidosis Activity and Severity Index (SASI) was the first outcome instrument proposed to measure cutaneous sarcoidosis severity and has been incorporated into clinical trials.8,9 SASI has been reported to be reliable9; however, its construct validity and other psychometric properties have not been reported.
In this study, we propose a novel instrument, the Cutaneous Sarcoidosis Activity and Morphology Instrument (CSAMI), designed to capture disease activity and morphologic type for use in clinical trials and prognostic studies. Our primary objectives were to assess intrarater and interrater reliability and convergent validity of CSAMI and SASI compared with the Physician's Global Assessment (PGA) as reference. Our secondary objectives were to evaluate the instruments' correlations with quality-of-life measures and the time required for their completion.
The study was approved by the University of Pennsylvania institutional review board and reported based on the Strengthening the Reporting of Observational Studies in Epidemiology statement.10 Written informed consent was obtained from all patients.
Eight dermatologists with experience diagnosing and managing cutaneous sarcoidosis, including 7 board-certified dermatologists (M.R., E.Y.C., E.J.K., A.S.P., J.T., C.C.V., and J.M.G.) and 1 dermatology chief resident (K.A.W.), were invited to complete this 1-day study in June 2012. All completed a training session with images of cutaneous sarcoidosis to become familiarized with the 3 outcome instruments and their scoring methods. Questions regarding the outcome instruments were addressed before evaluation of cutaneous sarcoidosis in patients.
Eleven patients were recruited from the cutaneous sarcoidosis clinic at the Hospital of the University of Pennsylvania via telephone. Eligible patients had clinical and/or pathological evidence consistent with the diagnosis of cutaneous sarcoidosis. Patients were selected purposively by the principal investigator (M.R.) to include a wide range of sarcoidosis presentation and severity. On the study day, all patients completed 3 surveys about the effects of cutaneous sarcoidosis on their health-related quality of life.
Patients were randomly divided into 2 groups. Physicians scored sarcoidosis in one group using CSAMI followed by SASI and the PGA; then they scored sarcoidosis in the other group using SASI followed by CSAMI and the PGA. Physicians were instructed to document their start and stop times for each instrument. Physicians rotated among individual patient rooms, rating sarcoidosis in each of the 11 patients using all 3 instruments, and then rerating the diseases in 3 patients. Rerating was performed based on patient availability. All but 1 patient underwent rerating, 3 patients rerated by 1 physician, 2 patients rerated by 2 physicians, 3 patients rerated by 3 physicians, and 2 patients rerated by 4 physicians.
CSAMI was developed by the principal investigator (M.R.) to capture disease activity and morphologic type after reviewing relevant literature on cutaneous sarcoidosis and outcome instruments in dermatology.11,12 Lessons learned through developing cutaneous scoring instruments for lupus, dermatomyositis, and pemphigus were incorporated into instrument design. Face validity, or the extent to which CSAMI seemed to represent its underlying construct of cutaneous sarcoidosis severity, was established by review of a preliminary instrument by multiple authors for its format and content (E.J.K., J.T., V.P.W. and J.M.G.) and piloted in the clinic (M.R., H.Y., E.J.K., and K.A.W.) with changes made to enhance clarity and usability.
The final CSAMI consists of 2 scores measuring disease activity and damage done by the disease (Figure). The Activity and Damage scales are considered separately to aid the instrument in detecting changes in disease activity, rather than remaining stable as a single conglomerate outcome as inflammatory activity subsides and chronic damage develops.11 Activity is scored based on inflammation, induration and/or depression, surface changes, such as scaling and ulceration, and area of involvement. Damage is scored based on dyspigmentation and scarring. Clinical signs are documented according to the worst affected lesion within each anatomical area and summed, with maximal score ranges of 0 to 165 for the Activity scale and 0 to 22 for the Damage scale. In addition, CSAMI assesses morphologic types of cutaneous sarcoidosis lesions, documenting a predominant type and all other types present. The instrument also examines the presence of lesion types that connote specific significance when present, including lupus pernio and erythema nodosum.
SASI was the first proposed outcome instrument for cutaneous sarcoidosis.8,9 SASI evaluates the following 4 features for each of the 4 facial quadrants and the nose: erythema, induration, and desquamation, each ranging from 0 (none) to 4 (very severe), and an area score ranging from 0 (0%) to 6 (90%-100%). Thus, SASI produces 5 separate sets of scores per patient. The Facial SASI score weighs these SASI components to provide a composite index for the face; however, it requires rescoring for the lower face and the nose. SASI has been previously modified and incorporated into clinical trials.8 Similar to that method, we modified the Facial SASI to simplify its computation: the sums of the erythema, induration, and desquamation scores for each quadrant of the face and the nose were multiplied by their respective area scores and then averaged with equal weight on all 5 regions. The maximal range of the modified Facial SASI scores is 0 to 72.
The PGA is a visual analog scale ranging from 0 (perfect health) to 10 (worst skin condition imaginable). This form of the PGA has been used to rate the overall impression of disease severity in instrument validation studies for dermatomyositis and pemphigus.13,14 Because no criterion standard instrument exists for cutaneous sarcoidosis against which we could evaluate criterion validity, we used the PGA to assess convergent validity, expecting positive correlations between CSAMI and the PGA and between the modified Facial SASI and the PGA in reflecting the overall level of disease severity.
Each patient completed the following 3 self-administered, health-related quality-of-life surveys: the revised 29-item Skindex (Skindex-29),15 Dermatology Life Quality Index (DLQI),16 and Sarcoidosis Health Questionnaire (SHQ).17 Skindex-29 and the DLQI are validated dermatology-specific quality-of-life instruments widely used in the literature. Skindex-29 surveys the domains of emotions, symptoms, and functioning, each ranging from 0 (no effect on quality of life) to 100 (effect always experienced).15 The DLQI is a 10-item survey on the impact of skin diseases, with scores ranging from 0 (no effect on patient's life) to 30 (extremely large effect).16 The SHQ is a 29-item validated sarcoidosis-specific survey assessing the impact of sarcoidosis involvement of multiple organ systems, summarized by a total score ranging from 1 (effect experienced all of the time) to 7 (none of the time).17 However, the SHQ contains only 1 question pertinent to the skin and may not capture the full effect on skin-related quality of life.
We summarized scores from each outcome instrument descriptively. Skewness and the kurtosis test were used to assess normality of score distributions. Reliability was analyzed using the intraclass correlation coefficient (ICC), a more robust measure of agreement than other statistics.18 Intrarater and interrater ICCs were calculated using 1- and 2-way random-effects models, respectively, and interpreted as less than 0.40 for poor; 0.40 to 0.75, fair to good; and greater than 0.75, excellent.18,19 Because the SASI provides multiple scores per patient, its intrarater and interrater reliability were calculated by each facial region of each patient.9 All other instruments, including the modified Facial SASI, were calculated by patient. SASI scores from 1 patient with no facial involvement were excluded to minimize bias in reliability analysis. Reliability of CSAMI morphologic types was analyzed using κ statistics and interpreted as 0 to 0.2 for slight agreement; 0.2 to 0.4, fair; 0.4 to 0.6, moderate; 0.6 to 0.8, substantial; and greater than 0.8, almost perfect.20- 22
Construct validity refers to the degree to which one measure correlates with another measure with which it theoretically should correlate. Convergent validity, a form of construct validity that compares against a similar construct of disease severity, was assessed for the correlation between the CSAMI and the PGA and between the modified Facial SASI and the PGA using the Spearman rank correlation (ρ value). Mixed-effects linear regression was used to confirm the linearity of these associations, adjusting for interrater and intrarater variations as random effects and PGA score as a fixed effect. Construct validity was also evaluated in terms of the correlation between mean instrument scores and quality-of-life metrics, using the Pearson product moment correlation (r value) or the Spearman rank correlation as appropriate.
Time required for instrument scoring was calculated from instrument start and end times, rounded up by the minute, and compared using 2-tailed t tests. Physicians did not calculate the CSAMI Activity and Damage scores or the modified Facial SASI score while the patients underwent evaluation; thus, the scoring time did not reflect time required for manual calculation. Statistical analyses were performed using commercially available software (STATA, version 12.1; StataCorp LP).
We planned to have 12 patients with 8 physician ratings per patient to detect an interrater ICC of 0.7 with 80% power, when the ICC is 0.4 under the null hypothesis, using an F test with a significance level of .05.
Eleven patients were available and participated in this study. Their mean (SD) age was 52.5 (7.6) years; 2 patients were male. Nine patients were African American and 2 were white. Sarcoidosis involvement was documented in a mean (SD) of 3 (1) organ systems, with skin and lung involvement seen in all patients. A wide spectrum of skin disease severity was represented as demonstrated by the range of PGA scores (Table 1). Patients' cutaneous morphologic type and anatomic area affected, as measured by CSAMI, and current treatments were shown in Table 2. CSAMI scores and SASI had positively skewed distributions, whereas the PGA scores were approximately normally distributed.
Excellent intrarater reliability was demonstrated with the CSAMI Activity, SASI components, modified Facial SASI, and PGA, whereas the CSAMI Damage scale had good intrarater reliability (Table 3). The CSAMI Activity scale demonstrated excellent interrater reliability; in contrast, the CSAMI Damage scale, SASI components, modified Facial SASI, and PGA had interrater reliability ranging from fair to poor.
The proportional overlap of selected morphologic types demonstrated substantial intrarater reliability (κ = 0.66 [95% CI, 0.47-0.84]) and moderate interrater reliability (0.46 [0.33-0.59]). The predominant morphologic type selected also showed substantial intrarater reliability (κ = 0.66 [95% CI, 0.35-0.90]) and fair interrater reliability (0.35 [0.23-0.50]). The presence of lupus pernio displayed substantial intrarater reliability (κ = 0.74 [95% CI, 0.46-1.00]) and fair interrater reliability (0.34 [0.15-0.55]). Erythema nodosum was rated as absent in all patients.
Convergent validity was assessed by correlating each instrument against the PGA as the reference measure, with significant correlations expected between similar constructs of disease severity (Table 3). The CSAMI Activity scale and modified Facial SASI showed moderate correlations with the PGA, whereas the CSAMI Damage scale showed a weak correlation. Mixed-effects regression modeling also demonstrated that a unit increase in the PGA score significantly predicted linear increases in the CSAMI Activity (regression coefficient β, 4.92 [95% CI, 3.30-6.54; P < .001]) and CSAMI Damage scores (0.59 [0.30-0.88; P < .001]) and modified Facial SASI (1.14 [0.78-1.51; P < .001]).
Construct validity was evaluated by comparing the disease severity instruments against health-related quality-of-life measures, with the expectation of positive correlations between the two. Mean (SD) of SHQ total score was 4.1 (1.0); Skindex-29 Emotions, Symptoms, and Functioning domain scores were 66.2 (21.0), 47.3 (20.3), and 43.0 (21.2), respectively. The median DLQI score was 5 (interquartile range, 2-7). Strong to moderate correlations were demonstrated between CSAMI Activity and Damage scales with the Skindex-29 Emotions and Functioning domains, respectively (Table 4). Moderate to weak nonsignificant correlations were found between CSAMI and the DLQI and SHQ total score, whereas weak to slight nonsignificant correlations were found between the modified Facial SASI and all quality-of-life measures.
The mean (SD) of CSAMI scoring time decreased significantly from 5.0 (1.7) to 3.5 (1.2) minutes from the first to the second ratings (paired t test, P < .001), whereas mean SASI scoring time decreased from 3.0 (1.0) to 2.4 (0.9) minutes, respectively (P = .02). Overall, CSAMI required a mean of 1.8 minutes longer to complete than SASI (4.5 vs 2.7 minutes [unpaired t test, P < .001]). Scoring time for PGA was not assessed.
Cutaneous sarcoidosis, like other chronic inflammatory dermatoses, may confer substantial morbidity and impairments in quality of life.23,24 However, the lack of an established and validated disease severity instrument hinders rigorous research in tracking sarcoidosis severity and evaluating therapeutic efficacy. The ideal outcome instrument should be reliable and accurate in scoring disease severity, simple to use in clinical practice, and sensitive to changes in disease course over time, with improvements in the instrument gained by iterative revisions.25- 27 In this study, we evaluated the reliability, convergent validity, construct validity, and practical applicability of the newly proposed CSAMI and the existent SASI in evaluating cutaneous sarcoidosis.
All 3 instruments demonstrated high intrarater reliability. Fair to poor interrater reliability of individual SASI component scores and modified Facial SASI are consistent with results from the original study by Baughman et al.9In contrast, the interrater reliability of CSAMI Activity scores were excellent and significantly higher than those of SASI, whereas the interrater reliability of CSAMI Damage scores were fair and comparable with that of SASI.
CSAMI Activity and Damage scales and modified Facial SASI correlated modestly with the PGA, with significant linear associations demonstrated. Given its poor interrater reliability, the PGA is an imperfect reference that may not be very useful in assessing cutaneous sarcoidosis activity per se. Nevertheless, because no established criterion standard exists against which criterion validity may be construed, convergence between CSAMI and modified Facial SASI and the PGA suggests that both instruments reflect physicians' global impression of disease activity. The CSAMI Damage scale does not correlate particularly well with the PGA, because the Damage scale is designed to capture residua and not active disease, and many physicians may use the PGA to measure disease activity.
Strong to moderate correlations between CSAMI and Skindex-29 domains provided preliminary evidence for construct validity, wherein objective evaluation using CSAMI correlated well with patients' subjective impression of disease impact. Given our small sample size, the analysis is underpowered to detect more modest correlations with other quality-of-life metrics. Nevertheless, point estimates of these correlations trended higher with CSAMI than with SASI. Larger studies to confirm the ability of CSAMI and/or SASI to predict quality-of-life impact are needed.
Both instruments were scored in less than 5 minutes, which is considered adequate for use in clinical trials and may be acceptable for routine clinical practice.28 Although CSAMI required longer to score than SASI, the time difference narrowed during repeated ratings.
CSAMI captured lesion morphologic types with substantial intrarater and moderate interrater reliability. Although prognostic implications of specific lesions like lupus pernio and erythema nodosum are widely recognized, those of other morphologic types are less well established.2 Previous studies have implicated lupus pernio with increased risks of sinus disease and bone cysts and subcutaneous lesions with increased risks of systemic disease.29 Documentation of morphologic types using CSAMI may facilitate studies on prognostic relationships between lesional morphologic types and systemic involvement as well as treatment response.
We believe that a cutaneous sarcoidosis instrument should include a physician rating of inactive lesions, including hyperpigmentation, with the caveat that given the protean nature of sarcoidosis, distinguishing inactive postinflammatory residua from active lesions may be challenging in some cases. However, persistent hyperpigmentation or scarring may be inelastic to therapy and significantly affect patients' quality of life; thus separately categorizing these lesions may be important for evaluating therapeutic response in clinical trials. CSAMI is designed to capture the impact of involved area size, without subjecting that metric to a multiplier that could magnify carried errors in the final numbers. Treatments often first affect lesion erythema or induration, whereas area may be slower to respond; thus we believe that incorporating area as a separate value in CSAMI may better reflect cutaneous sarcoidosis activity than as a multiplier, as in SASI.
Our study should be reviewed in light of its limitations. Recall bias on intrarater reliability cannot be excluded given the relatively short time elapsed between the first and second ratings within a single afternoon, but it was minimized because the physicians and the patients were not told about the rescoring session at the study onset. Although our patients were selected to represent a wide spectrum of cutaneous sarcoidosis activity, the smaller-than-planned sample size limited our data's external validity, particularly for patients with different clinical presentations, disease severity, and/or treatment history. All physician participants were dermatologists, so our results may not be generalized for instrument use by nondermatologists. Future studies are necessary to confirm the instruments' correlations to quality-of-life measures, show their sensitivity to change, and provide meaningful interpretation for minimal clinically important differences.3 Iterative revisions of the instruments, in particular the less reliable subscales, may optimize their psychometric properties and refine their ability to categorize cutaneous sarcoidosis severity.
In conclusion, this study provided psychometric validation of CSAMI and SASI in patients with cutaneous sarcoidosis. CSAMI has demonstrated reasonable reliability, convergent validity, and clinical acceptability and captured a wide range of body surface and cutaneous morphologic types. Sensitivity to change and preliminary evidence of construct validity between CSAMI and quality-of-life measures should be further examined. CSAMI should be considered as an outcome instrument for the evaluation of disease severity and documentation of morphologic types of lesions.
Correspondence: Misha Rosenbach, MD, 2 Maloney Bldg, 3600 Spruce St, Philadelphia, PA 19104 (firstname.lastname@example.org).
Accepted for Publication: December 20, 2012.
Author Contributions: Dr Rosenbach and Mr Yeung contributed equally to the work, have full access to all of the data in the study, and take responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Rosenbach, Yeung, Werth, and Gelfand. Acquisition of data: Rosenbach, Yeung, Chu, Kim, Payne, Takeshita, Vittorio, Wanat, and Gelfand. Analysis and interpretation of data: Rosenbach, Yeung, Payne, and Gelfand. Drafting of the manuscript: Yeung. Critical revision of the manuscript for important intellectual content: Rosenbach, Yeung, Chu, Kim, Payne, Takeshita, Vittorio, Wanat, Werth, and Gelfand. Statistical analysis: Yeung and Gelfand. Administrative, technical, or material support: Rosenbach, Yeung, Kim, Payne, Vittorio, Wanat, and Gelfand. Study supervision: Rosenbach and Gelfand.
Conflict of Interest Disclosures: Dr Rosenbach led the development of the Cutaneous Sarcoidosis Activity and Morphology Instrument.
Funding/Support: This study was supported by training grant T32-AR07465 (Mr Yeung) and grant K24-AR02207 (Dr Werth) from the National Institutes of Health; a National Psoriasis Foundation Fellowship (Dr Takeshita); the Office of Research and Development, Biomedical Laboratory Research and Development, Veterans Health Administration; and a Medical Dermatology Career Development Award from the Dermatology Foundation (Dr Rosenbach).
Role of the Sponsors: The sponsors had no role in the design and conduct of the study; in the collection, analysis, and interpretation of data; or in the preparation, review, or approval of the manuscript.
Additional Contributions: Rosemary Attor, BA, provided administrative support during the research day.
This article was corrected for errors on May 30, 2013.