Development and Validation of the Rasch-Built Overall Amyotrophic Lateral Sclerosis Disability Scale (ROADS)

Key Points

Question  By combining clinical expertise with mathematically rigorous Rasch methodology, can a new self-reported amyotrophic lateral sclerosis disability scale be developed with improved reliability and responsiveness compared with the ordinal scale in current use?

Findings  In this study of 243 individuals, the 28-question Rasch-Built Overall Amyotrophic Lateral Sclerosis Disability Scale was created and had improved item targeting and high test-retest reliability.

Meaning  This new Rasch-Built disability scale may serve as an easily accessible outcome measure that could improve the efficiency of amyotrophic lateral sclerosis clinical trials and serve as an informative tool in the clinic setting.

Importance  A new outcome measure for overall disability level with improved responsiveness is needed for amyotrophic lateral sclerosis (ALS) clinical trials.

Objective  To describe the creation and development of a new self-reported ALS disability scale with improved item targeting and psychometric properties that used a mathematically rigorous Rasch methodology.

Design, Setting, and Participants  A preliminary ALS disability questionnaire with 119 questions was created based on literature review, clinical judgement of an expert panel, and patient input. Patients with ALS were recruited from January 2017 to June 2019 from the Emory University and Atlanta VA Medical Center ALS clinics, both in Atlanta, Georgia, during regularly scheduled clinic appointments to complete the draft questionnaire and standard ALS outcome measures. All consecutive patients seen at the Emory University and Atlanta VA Medical Center ALS clinics during the recruitment period with a diagnosis of ALS who were able to provide informed consent were invited to participate in the study. Rasch analyses were performed, and items were systematically removed based on missing data, model fit, disordered thresholds, item bias, and clinical judgment. A total of 509 patients with ALS were seen at the 2 sites during the recruitment period, and 264 patients provided informed consent.

Interventions  Participants completed the draft Rasch questionnaire and the revised Amyotrophic Lateral Sclerosis Functional Rating Scale (ALSFRS-R).

Main Outcomes and Measures  Rasch analyses and standard scale metrics were performed to create the new scale, and Rasch analyses were performed on the ALSFRS-R for comparison.

Results  Overall, 243 participants with ALS completed the draft questionnaire, and 230 participants were included for Rasch analyses. The mean (SD) age for study participants was 61.9 (11.1) years, 146 (60.1%) were men, and site of onset was 23.0% bulbar (n = 56), 36.2% upper extremity (n = 88), and 39.5% lower extremity (n = 96). A 28-question Rasch-Built Overall ALS Disability Scale (ROADS) was constructed with each item scored 0, 1, or 2. The ROADS fulfilled Rasch model requirements, demonstrated improved item targeting compared with the ALSFRS-R, and had test-retest reliability of 0.97. Individual question fit statistics demonstrated infit values from 0.68 to 1.37 and outfit values from 0.66 to 1.43. The difference between the empirical variance explained by the measures and the modeled variance was 0.1%. The ALSFRS-R violated Rasch model expectations and demonstrated disordered thresholds for 9 of 12 questions; 13 of 48 answer choices on the ALSFRS-R were never the most probable answer choice for any overall disability level.

Conclusions and Relevance  In this study, the 28-question, self-reported ROADS, which is linearly weighted, had improved item targeting compared with the ALSFRS-R, had high test-retest reliability, and was validated. ROADS may serve as a valuable and easily accessible outcome measure for use in ALS trials and in the clinic with improved responsiveness compared with the ALSFRS-R.

The revised Amyotrophic Lateral Sclerosis Functional Rating Scale (ALSFRS-R) is commonly used as the primary outcome measure for contemporary clinical trials aiming to slow disease progression in patients with amyotrophic lateral sclerosis (ALS).^1-6 The ALSFRS-R is an ordinal rating scale that includes 12 questions, each rated 0 through 4, that assesses the overall functional status of the patient with ALS.⁷ Strengths of this scale include questions with obvious face validity and a reasonable, although incomplete, correlation of the ALSFRS-R sum score with other more difficult and costly outcome measures such as measures of strength, electrophysiologic measures, or survival.^8-10 The ALSFRS-R is accessible for widespread use and can be administered over the telephone or online.^11,12

However, the ALSFRS-R also has problems that hinder its clinical and research utility. Analysis of an ALS pooled clinical trial database (PRO-ACT¹³) has shown that 25% of patients receiving placebo or a failed therapeutic agent showed no decline on the ALSFRS-R over a 6-month period.¹⁴ This striking lack of scale responsiveness dilutes the ability to detect a treatment effect in clinical trials if one is in fact present, particularly when many ALS treatment trials test agents designed to slow disease progression and use a 6-month treatment period. The ALSFRS-R also contains questions that show improvement due to changes in symptom management or due to changes in behavior. For example, if a patient receives symptomatic treatment to reduce sialorrhea, score improvements can occur.¹⁵ As another example, patients lose points on the ALSFRS-R as they increase use of noninvasive positive pressure ventilation, and score improvements would occur if a patient chooses to discontinue noninvasive ventilation even if ventilatory muscle strength is declining. Despite the fact that ALS is an almost universally progressive disease, 14% of patients in a pooled ALS clinical trial database experienced a 180-day period of improved ALSFRS-R slope,¹⁴ which does not match the biologic reality of ALS disease course or the findings on objective outcome measures.^10,16,17

Modern test theory techniques, such as Rasch methodology, can be used to create scales that outperform traditional ordinal scales such as the ALSFRS-R, which are created with classic theory techniques.^18-21 While both classic and modern test theory approaches rely on assessment of construct validity, reliability, and internal consistency, classic methods rank data based on whether a value is greater or less compared with another value, but the distance between each number cannot be quantified and is expected to be unequal between categories.¹⁸ Rasch-built scales are linearly weighted, meaning that a 1-point change is a consistent measurable unit across the scale, an important feature when a scale sum score is being used as an outcome measure. The ALSFRS-R is not linearly weighted. As an example, the score of a patient with ALS will decline by 3 points on the ALSFRS-R if there is a change from climbing stairs normally to holding the handrail while climbing stairs, and a 3-point loss also occurs if a patient goes from normal dressing and hygiene to needing an attendant for self-care. This discrepancy shows that a 3-point loss in the total score can indicate a slight decline in function or a drastic loss of function; 1 point is not a measurable unit of function on an ordinal scale such as the ALSFRS-R.

Rasch-built scales are also unidimensional, meaning that all questions are measuring the same domain, and thus the sum score is a meaningful measure of overall status. The ALSFRS-R purports to measure functional ability, but it also includes respiratory questions that are not a measure of function.²² For example, a patient who is compliant with noninvasive ventilation will have a lower score on the ALSFRS-R compared with a patient with identical vital capacity who does not comply with noninvasive ventilation. Prior Rasch analyses of the ALSFRS-R have objectively demonstrated the lack of unidimensionality on the ALSFRS-R.^23,24

Rasch-built scales also allow for detailed assessment of item targeting, meaning that the difficulty level of each item is quantified to allow careful selection of items capturing a broad range of ability levels. Improved item targeting is likely one of the reasons that Rasch-built scales have been shown to be more responsive than classically built ordinal scales when assessing outcomes in other neurologic diseases,^25,26 something that is direly needed to conduct more efficient ALS clinical trials. The goal of this study was to create and validate a Rasch-Built Overall ALS Disability Scale (ROADS), a publicly available tool that is available at no charge,²⁷ for use as a clinical outcome measure in patients with ALS.

Patients with a diagnosis of ALS at all stages of disease who were able to provide informed consent were recruited consecutively from the Emory University and the Atlanta VA Medical Center ALS clinics, both in Atlanta, Georgia, from January 2017 to June 2019 during regularly scheduled clinic appointments. The institutional review boards of both institutions approved the study. Written informed consent was obtained from all participants. Questionnaire responses were provided by the patient, but caregivers or staff members were allowed to mark answer responses for patients needing assistance.

At the time of the clinic visit, study participants completed the first questionnaire containing 119 questions along with standard ALS outcome measures, the ALSFRS-R and vital capacity spirometry testing. The ALSFRS-R was completed by trained research coordinators with certification for scale administration, and spirometry was performed either by a trained research coordinator with spirometry certification or by a licensed respiratory therapist (V.J., M.P, and nonauthors). Vital capacity percentage of predicted value was calculated automatically by the spirometry devices using Knudson criteria.²⁸ On completion of the clinic visit, participants were given another blank copy of the 119-question questionnaire to complete at home within 2 to 7 days to return to study team by mail for assessment of test-retest reliability.

A preliminary ALS disability questionnaire was created and modified based on literature review, clinical judgment of an expert panel, and patient input. Literature review included existing scales that assess function or disability specifically in an ALS population,^7,29 scales assessing function or disability that have been validated in an ALS population^30,31 and relevant scales of function or disability for other neurologic diseases that were built or modified with Rasch methodology.^20,21,32-34 The expert panel included 5 neurologists (C.N.F., R.B., C.Q., J.R., and J.D.G), 1 physical therapist (D.B.), and 1 speech-language pathologist (K.H.K.), all with ALS subspecialty expertise and in-depth experience working with patients with ALS in tertiary academic referral centers. The goal of expert panel and patient review in developing the initial question bank was to ensure creation of a comprehensive question bank assessing the full range of ALS disability levels and phenotypes. The initial questionnaire contained 119 questions related to daily activities for a broad-based ALS population. For each question, item response options were 0, unable to perform; 1, abnormal: able to perform but with difficulty; 2, normal: able to perform without difficulty; or 3, not applicable. The written instructions provided additional clarification that answers should be scored based on how the task is usually performed; items should be scored as 2 (normal) if the task was performed as easily and as quickly as it was before having ALS symptoms, and items should be scored as 1 (abnormal) if it was harder to perform the task, the task takes more time or effort, or the task is performed with the assistance of a device or another person.

Questionnaire items with more than 10% missing values were removed, and participants were excluded from analysis if more than 10% of items were missing from their questionnaire. Items marked as 3 (not applicable) were considered as missing. Rasch analyses were performed on remaining responses with Winsteps software (version 4.4.4) using the partial credit model, allowing each item to have its own response structure.³⁵ Category probability curves should demonstrate that for the most severe overall disability level, the lowest item response selection is the most probable, and as overall disability level improves, each item response answer sequentially becomes the most probable response (Figure 1). When this does not occur, an item demonstrates disordered thresholds. Category probability curves were examined for each question, and any question with disordered thresholds was removed.

Differential item functioning occurs when 2 or more groups of respondents of the same ability level respond differently to an item based on a factor other than disability. Differential item functioning was examined on the basis of age and sex, and any item demonstrating significant bias was removed.

Analyses of dimensionality, individual fit statistics, and overall model fit statistics were assessed as questions were systematically removed from the questionnaire to create the final scale. Principal components analysis on the standardized residuals was used to assess dimensionality, and variance explained by the measured construct of greater than 50% was considered sufficient for unidimensionality.³⁵ Fit statistics examined include infit or the inlier-sensitive fit, outfit or the outlier-sensitive fit, and mean-square statistics showing the size of the randomness.³⁵ A correlation matrix was examined to assess local dependency, or the association of each question to other questions on the scale, to avoid interdependent questions.

The overall disability score was compared among patients of different sites of onset to attain the clinically based goal of balancing assessment of bulbar, upper extremity, and lower extremity function. The expert panel reviewed the final questionnaire to evaluate item difficulty order, content validity, and overall clinical utility of the complete scale. The expert panel removed any questions with insufficient face validity or that were expected to demonstrate improvement in score due to factors other than ALS reversal.

Per prior Rasch literature, a sample size of 250 participants provides 99% confidence with stable item calibration with ±0.5 logits, allowing for a stable model.^20,36 External construct validity was evaluated by assessing correlations of the total ROADS score to the total ALSFRS-R score and to vital capacity percentage of predicted value completed on the same day. Rasch analyses were also performed on the ALSFRS-R using an anchoring approach³⁷ with common ROADS items to allow direct comparison of scale metrics and item targeting. Anchoring sets the Rasch measure for common questions seen on both scales to the same value, allowing direct comparison of item difficulty measures along the same linear ruler. Specifically, for items that appear on both the ROADS and the ALSFRS-R, the logit difficulty measures that were calculated through Rasch analyses of the ROADS were preset to identical values when performing Rasch analyses on the ALSFRS-R. Test-retest reliability was assessed by examining correlation coefficients for the overall ROADS 28-question sum score for 2 questionnaires completed 2 to 7 days apart. Analysis began July 2019.

A total of 509 patients with ALS were seen at the 2 sites during the recruitment period. An unselected group of 264 patients with ALS provided informed consent for study participation, and 243 completed at least one 119-item questionnaire. Characteristics of these 243 patients are included in Table 1. Thirteen study participants were removed from the data set prior to Rasch analyses owing to more than 10% missing answers, and 7 questions were removed owing to more than 10% missing responses. A total of 67 participants completed a second 119-item questionnaire within 2 to 7 days for the test-retest analyses.

Category probability curves were examined for each remaining question, and 4 questions (drive a car, lick an envelope or stamp, run, and whisper quietly) were removed owing to disordered thresholds, meaning that at least 1 of the answer choices was never the most probable response for any overall disability level (Figure 1B). No items demonstrated significant differential item functioning on the basis of age when comparing participants above and below the mean age for this data set. Two items (read a newspaper or book and watch television) demonstrated differential item functioning on the basis of sex and were removed.

For the remaining questions, the panel of clinical experts removed any items with insufficient face validity or items that were expected to improve owing to factors other than ALS reversal. Additional items were systematically removed based on individual item fit statistics, overall model fit, and local dependency based on the item correlation matrix, leaving the final 28-item ROADS (Figure 2) with each item scored as 0, 1, or 2. This final scale met Rasch model expectations. Item-measure correlations were all positive, ranging from 0.53 to 0.74, providing evidence of content validity. A negative item–correlation measure would suggest a negative correlation between disability and item responses, which can happen with reverse coding or improperly worded questions.

Infit and outfit statistics indicate how accurately the data for each question fit the Rasch model, with a value of 1 indicating a perfect fit, and values between 0.5 to 1.5 considered productive for measurement.^38,39 Individual question fit statistics for the ROADS demonstrated that infit values on each item ranged from 0.68 to 1.37, and outfit values ranged from 0.66 to 1.43, indicating appropriate item fit for all selected questions.^38,39 Principal components analysis on the standardized residuals was used to assess dimensionality, which is an assessment of whether questions are measuring the same domain, in this case disability. A variance explained by the measured construct of greater than 50% is considered sufficient for unidimensionality,³⁵ and for the ROADS variance explained by the measured construct was 58.2%. The strength of the first residual contrast showed insufficient association with the total measurement structure, meaning that this group of questions may not have segregated from the others enough to warrant their classification as a different dimension. The difference between the empirical variance explained by the measures and the modeled variance was 0.1%, indicating a close match between what was observed in the ROADS and what was expected based on the Rasch model.

Test-retest reliability on the 28 ROADS questions for the 67 participants who completed 2 questionnaires was excellent with an intraclass correlation coefficient of 0.97. In a prior study, interrater reliability for the ALSFRS-R has been reported as 0.93, while intrarater reliability has been reported as 0.95.¹¹ External construct validity was good between the ROADS and the ALSFRS-R with a correlation of 0.82. Vital capacity percentage of predicted value and ROADS had a moderate correlation of 0.57, matching the expected association between disability level and ventilatory status in patients with ALS as shown in prior clinical studies.⁷ The correlation between the ALSFRS-R sum score and the vital capacity percentage of predicted value was 0.65.

Rasch analyses were performed on the ALSFRS-R data set from the same patient cohort (Figure 3). ROADS items use a knife and fork (measure = 0.16) and roll over in bed (measure = −0.03) were used as anchors for ALSFRS-R questions 5 (cutting food) and 7 (turning in bed), meaning that the logit measures for the ROADS questions were preset to 0.16 and −0.03 for the corresponding ALSFRS-R questions when Rasch analyses were performed on the ALSFRS-R. The ROADS targeted a broader range of disability levels compared with the ALSFRS-R; item difficulties spanned 5.9 logits for the ROADS (range, −2.97 to 2.93 logits) but only 2.27 logits (range, −1.32 to 0.95 logits) for the ALSFRS-R.

Examination of category probability curves of the ALSFRS-R (Figure 1C) showed that 9 of 12 questions demonstrated disordered thresholds. Thirteen of 48 answer responses on the ALSFRS-R were never the most probable answer choice for any overall functional rating ability.

Overall disability or functional rating score was not different between patients with bulbar vs upper extremity vs lower extremity symptom onset either on the ROADS or the ALSFRS-R. For the ROADS, the mean (SD) raw sum score was 28.5 (11.8) for patients with bulbar onset, 26.5 (8.6) for patients with lower-extremity onset, and 25.5 (10.0) for patients with upper-extremity onset. For the ALSFRS-R, the mean (SD) sum score was 26.7 (9.0) for patients with bulbar onset, 28.5 (8.3) for patients with lower-extremity onset, and 27.8 (9.8) for patients with upper-extremity onset.

As logit units are not intuitive, Table 2 provides a conversion from logit units to normed whole numbers provided by WINSTEPS software, ranging from 0 to 146. While the raw sum score is not linearly weighted, normed whole number scores are linearly weighted, meaning that a 1-point change is a measurable and consistent unit of disability across the entire scale, and a 2-point change reflects twice the disability level compared with a 1-point change.

We successfully created and validated ROADS, which is expected to overcome many of the limitations seen with the currently used ordinal ALSFRS-R. While there are always limitations to subjective self-reported outcome measures, using Rasch methodology brings a mathematical rigor to this type of scale that is not present in standard ordinal scales. The ROADS demonstrates a wider range of item targeting compared with the ALSFRS-R, meaning that while both scales contain questions targeting respondents of average disability level, only the ROADS contains questions targeting respondents with higher and lower levels of overall disability, and thus the ROADS is expected to be able to better differentiate levels of overall disability for respondents with very high or very low levels of overall disability. The breadth of item targeting is expected to improve scale responsiveness, meaning that the scale is more likely to detect change when change has actually occurred. The performance of the ROADS in terms of responsiveness will need to be tested in future longitudinal studies for confirmation.

Unlike the ALSFRS-R, the ROADS is linearly weighted, meaning that a 1-point change in the overall normed score captures a measurable unit of disability that is consistent across the entire scale. A 1-point change in the ordinal ALSFRS-R is not a quantifiable measure of function and can represent a small or large loss of functional ability depending on the question. Dimensionality analyses of the ROADS confirm that each question is measuring the same domain (disability), while the ALSFRS-R violates unidimensionality,^23,24 meaning that changes in the ALSFRS-R might not be associated with functional status. Additionally, ROADS questions were carefully selected by the panel of clinical experts to avoid questions that may improve in the absence of disease reversal¹⁴ or owing to factors not associated with ALS. These limitations are highly relevant given that drugs are approved or disapproved for slowing disease progression in ALS based on ALSFRS differences of only 2 to 3 points.^3,40

Selection of an appropriate primary outcome measure is vital for the success and efficiency of clinical trials. Creation of an improved outcome measure to quantify overall disability level, a highly relevant and important clinical outcome, is particularly important for ALS where there is no criterion standard to measure disease progression, there is no objective measure of overall disability or functional status, and current candidate biomarkers are still in exploratory stages. Even if biomarkers of disease activity are validated in the future, clinically meaningful outcome measures will still remain relevant for patients and clinicians and are strongly encouraged by the US Food and Drug Administration for use in clinical trials.⁴¹

Future studies are planned to examine the longitudinal performance of the ROADS, assess correlation of ROADS and survival, and examine predictive features of the scale. Ongoing studies are also planned to determine test-retest reliability for telephone-administered scales and scales completed by live-in caregivers.

In the ROADS instructions, patients are asked to score a task as 2 (normal) if the task is performed as easily and as quickly as it was before having ALS symptoms. These instructions introduce the risk of recall errors if patients are uncertain of prior task performance. Although the ALSFRS-R was not initially designed to compare current performance with pre-ALS performance, the scale as it is currently administered for ALS clinical trials does include these explicit instructions,^42,43 and thus the ALSFRS-R also has this same risk of recall errors.

The 28-question, self-reported ROADS was successfully created and validated. This scale is linearly weighted, has improved item targeting compared with the ALSFRS-R, and has high test-retest reliability. The ROADS can serve as a valuable and easily accessible outcome measure for use in ALS trials and in the clinic.

Corresponding Author: Christina N. Fournier, MD, MSc, Emory University, Department of Neurology, 101 Woodruff Cir, Ste 6000, Atlanta, GA 30022 (cfourni@emory.edu).

Accepted for Publication: November 1, 2019.

Published Online: December 30, 2019. doi:10.1001/jamaneurol.2019.4490

Author Contributions: Dr Fournier had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Fournier, Bedlack, Russell, Beckwith, Kaminski, Tyor, Hertzberg, Glass.

Acquisition, analysis, or interpretation of data: Fournier, Bedlack, Quinn, Hertzberg, James, Polak, Glass.

Drafting of the manuscript: Fournier, Quinn, Russell, Tyor, Polak, Glass.

Critical revision of the manuscript for important intellectual content: Fournier, Bedlack, Russell, Beckwith, Kaminski, Tyor, Hertzberg, James, Glass.

Statistical analysis: Fournier, Hertzberg.

Obtained funding: Fournier.

Administrative, technical, or material support: Bedlack, Kaminski, James, Polak.

Supervision: Fournier, Bedlack, Quinn, Tyor, Hertzberg, Polak, Glass.

Conflict of Interest Disclosures: Dr Bedlack reports grants from ALS Association, Motor Neuron Disease Association, Cytokinetics, Orion, and Ultragenyx and personal fees from ALS Association, Mallinkrodt, Biohaven Pharmaceutical, ITF Pharma, Brainstorm Cell Therapeutics, Biogen, New Biotic, Cytokinetics, and Woolsey Pharmaceuticals outside the submitted work. Dr Quinn reports personal fees for serving on the advisory boards of Acceleron Pharma and Amicus Therapeutics outside the submitted work. Dr Tyor reports grants from Atlanta VA Medical Center during the conduct of the study. No other disclosures were reported.

Funding/Support: This study was funded by the US Department of Veterans Affairs Office of Research and Development (grant IK2CX001595-02; Dr Fournier).

Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.