Validation of Brief Screening Tools to Identify Impaired Driving Among Older Adults in Australia

Key Points español 中文 (chinese)

Question  Are brief off-road screening measures sufficiently sensitive and specific to identify older drivers who will not pass an on-road driving test in Australia?

Findings  This prognostic study including 560 drivers aged 63 years and older found that off-road screening tests could identify older drivers who would not pass an on-road driving test with 77% sensitivity and 82% specificity. A combination of measures drawing from multiple skill domains provided the best prediction.

Meaning  These findings suggest that brief off-road screening tests could be a cost-effective, objective tool to screen older drivers to determine who might be an unsafe driver and to indicate referral for an on-road driving test.

Importance  There is an urgent need to develop evidence-based assessments to identify older individuals who may be unsafe drivers.

Objective  To validate 8 off-road brief screening tests to predict on-road driving ability and to identify which combination of these provides the best prediction of older adults who will not pass an on-road driving test.

Design, Setting, and Participants  This prognostic study was conducted between October 31, 2013, and May 10, 2017, using the criterion standard for screening tests, an on-road driving test, with analysis conducted from August 1, 2019, to April 2, 2020. A volunteer sample of older drivers was recruited from community advertisements, rehabilitation and driver assessment clinics, and an optometry clinic in Canberra and Brisbane, Australia.

Exposures  Off-road driver screening measures, including the Useful Field of View, DriveSafe/DriveAware, Multi-D battery, Trails B, maze test, Hazard Perception Test, DriveSafe Intersection test, and 14-item Road Law test.

Main Outcomes and Measures  Classification as unsafe on a standardized 50-minute on-road driving assessment administered by a driving instructor and an occupational therapist masked to the participant’s clinical diagnosis and off-road test performance.

Results  A total of 560 drivers aged 63 to 94 years (mean [SD] age, 74.7 [6.2] years]; 350 [62.5%] men) were assessed. Logistic regression and receiver operating characteristic analyses indicated the area under the curve was largest for a multivariate model comprising the Multi-D, Useful Field of View, and Hazard Perception Test, with an area under the curve of 0.89 (95% CI, 0.85-0.94), sensitivity of 80.4%, and specificity of 84.1% for predicting unsafe drivers. The Multi-D battery was the most accurate individual assessment and had an area under the curve of 0.85 (95% CI, 0.79-0.90), sensitivity of 77.1%, and specificity of 82.1%. The multivariate model had sensitivity of 83.3% and specificity of 91.8% in the cognitively impaired group and sensitivity of 87.5% and specificity of 70.8% in the visually impaired group.

Conclusions and Relevance  These findings suggest that off-road screening tests can reliably identify older drivers with a strong probability of failing an on-road driving test. Implementation of these measures could enable better targeting of resources for managing older driver licensing and support injury prevention strategies in this group.

While road safety overall is increasing with safer vehicles, ongoing demographic changes have led to sharp increases in the prevalence of road trauma involving older drivers. In Australia, road users aged 65 years and older accounted for 14.9% of road deaths in 2009, increasing to 21.3% in 2018.¹ Hospitalizations for injuries have also increased at a higher rate among older road users than younger road users, with similar statistics reported for the US.^1,2 Cognitive impairment and visual impairment are the 2 functional domains that have the largest effect on safety among older drivers and arguably pose the greatest difficulty for clinicians in terms of evaluation of driving risk.^3-7 An on-road driving test is the criterion standard for evaluating the safety of an older driver but can be expensive and time consuming. Screening tools of driver skills validated against an on-road test provide an urgently needed alternative.

Several off-road screening tests have been developed to assist with clinical decision-making for older drivers in the US and Australia.^8-12 However, limited data are available on the validity of most of these tests, and to our knowledge, they have never been compared within the same study on the same group of participants who have been assessed using the same on-road test. Our study addresses this gap in the literature, with a large sample of older drivers who underwent an on-road driving test.

We aimed to prospectively validate 8 brief measures used clinically or in research settings to identify drivers who would be classified as unsafe using an on-road driving assessment. Analysis of all measures in the single sample allows their relative predictive accuracy to be evaluated. We also sought to identify the optimal combinations of these measures to provide reliable off-road information for clinicians. We evaluated these measures in subgroups of participants with cognitive and visual impairment.

The study protocol was approved by the Human Research Ethics Committees of the Australian National University, Australian Capital Territory Government Health Directorate, and Queensland University of Technology. Written informed consent was obtained from all participants. This study is reported following the Standards for Reporting of Diagnostic Accuracy (STARD) reporting guideline (eTable 1 in the Supplement).

The Driving Aging Safety and Health project was undertaken in the Australian cities of Canberra and Brisbane. In Canberra, community dwelling drivers aged 65 years and over with a current valid license were recruited through advertisements in newspapers, community groups, and general practice and health clinics between October 31, 2013, and May 10, 2017. They were also recruited into the sample from the Australian Capital Territory Health Disability and Rehabilitation Service (DARS), which assesses drivers who are medically fit to drive but referred owing to concerns about driving safety, often in relation to cognitive impairment. Clients of DARS whose licenses were revoked were excluded from participation. Participants in Canberra completed a neuropsychological battery to identify mild cognitive impairment (MCI), as defined by Anstey et al.¹³ Participants in Brisbane aged 60 years and older with eye disease were recruited through the Queensland University of Technology Optometry Health Clinic and completed a full eye examination, including central and peripheral visual function testing.

Prevalence of unsafe driving was expected to be approximately 30.0% in older drivers with MCI,¹⁴ 35.0% in older drivers who were visually impaired,¹⁵ and 20.0% in the general population of older drivers.¹² Study sample size was planned assuming the sensitivity of Multi-D¹² to detect unsafe driving to be between 80.0% and 91.0%. Assuming 80% power and 5% critical error, a minimum of 67 drivers who would be considered unsafe were required per group; therefore, we planned required total samples of 223 participants with MCI, 191 participants who were visually impaired, and 335 participants without clinical impairment.

Recruitment via the electoral role was planned to obtain a representative sample of older drivers for the comparison groups but was not possible owing to prevailing policies that classified the study as nonmedical.

Assessments were conducted at either the Australian National University in Canberra or Queensland University of Technology in Brisbane. Questionnaires on driving habits, medical conditions, falls, and instrumental activities of daily living were administered.¹⁶ Off-road driver screening assessments were conducted by a trained research assistant prior to the participant undergoing an on-road driving test. A standardized research on-road driving test was administered by a driver-trained occupational therapist and qualified driving instructor masked to the results of the off-road driver screening assessments (described elsewhere¹³). The outcomes of the off-road screening assessments were not available to those conducting the on-road test.

Eight different tests that have been reported in the literature to screen for unsafe driving or crashes in older adults were included in the laboratory assessment: (1) Trail Making Test B from the Halstead-Reitan Battery of neuropsychological tests¹⁷; (2) the Useful Field of View (UFOV) subtest 2, a computer-based test of visual processing speed and divided attention with high reliability and validity demonstrated in other large studies to predict crash risk^10,18; (3) DriveSafe/DriveAware,⁹ a computer-based commercially available clinical screening tool for driving safety¹⁹ (the DriveSafe component of this test was used, which requires participants to view each of 15 images of the same roundabout intersection for 3 seconds, then verbally report details about the position and direction of travel of pedestrians and vehicle in the image)⁹; (4) DriveSafe Intersections, an additional multiple-choice questionnaire that assessed knowledge of road law; (5) the Hazard Perception Test (HPT),²⁰ a computer-based test requiring participants to view each of 20 video clips of 15 to 40 seconds’ duration depicting a traffic conflict and respond via touchscreen when they observed a potential traffic hazard in the video, which has been previously validated against self-reported crashes in older adults⁸; (6) the 14-item Road Rules and Road Craft test (Road Law), which comprises 14 questions about road safety and has been correlated with on-road test performance in clinical and research samples¹¹; (7) the Multi-D, a computer-based test that includes 3 subtests (choice color reaction time [requiring hand and foot responses and inhibition of responses], a test of sensitivity to central visual motion [using random dot stimuli presented at 3.2 m], and a test of balance or postural sway [using a sway meter to measure body displacement at the level of the waist]) and has been associated with on-road driving performance^12,21; and (8) the Occupational Therapy—Driver Off Road Assessment test,¹¹ a pen-and-paper maze test for which time taken to complete the maze has been associated with driving performance, similar to other maze tests.^22,23

At each site, within 3 months of the off-road assessment, participants completed a 50-minute open road assessment in an automatic vehicle with dual brake controls fitted. This followed a standard, validated protocol and scoring procedure that has been described previously.^24,25 Drivers’ safety was rated by an occupational therapist on a scale of 1 to 10, with a score of 1 to 3 indicating serious errors, unsafe driving behavior, and low likelihood of passing a driving test, while a score between 4 and 10 indicated a higher level of safety and likelihood of passing a driving test. Per our previous research, the occupational therapist and driving instructor were not involved in the administration of the off-road screening tests and were not informed of any results from these tests and hence were masked.^12,13,26 Interrater reliability of the safety rating was high (intraclass correlation = 0.94 [95% CI, 0.93 to 0.95]). Further details are provided in the eAppendix in the Supplement.

In Canberra, participants were included in the cognitive impairment subgroup if they were reviewed by the DARS clinic, had a known dementia diagnosis, had an Mini-Mental State Examination score less than 23, or met general criteria for MCI. A validated psychometric approach was used to classify participants²⁷ with MCI (eAppendix and eTable 2 in the Supplement).²⁸ Participants were classified as having MCI if they demonstrated subjective memory concern (Memory Complaints Questionnaire MAC-Q score >24), mild impairment (between 1 to 2 SDs below reference ranges) on 1 or more of 6 neurocognitive domains based on a battery of 11 cognitive measures (eTable 2 and eTable 3 in the Supplement), no or minimal self-reported impairment of instrumental activities of daily living due to cognition, and no evidence of dementia.¹³

In Brisbane, participants from any of the recruitment sources were classified as having visual impairment from any form of eye disease if they demonstrated any combination of central vision loss (assessed by visual acuity and/or contrast sensitivity), peripheral vision loss (defined as the presence of any significant visual field defects in either eye as defined by automated perimetry), or confirmed diagnosis by an eye care practitioner or evidence of ocular disease detected during the eye examination.

To conduct cross-sectional validation and comparison of the off-road screening measures against on-road safety, we used logistic regression to evaluate each measure against the on-road safety rating (safe: rating >3 vs unsafe: rating ≤3) adjusting for site (Canberra or Brisbane). We used this logistic regression to obtain a receiver operating characteristics (ROC) analysis of each measure. We obtained optimal cutoff points based on the Youden Index to estimate sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy of each screening measures based on this cutoff. We repeated these analyses on subgroups with eye disease and MCI to evaluate the validity of the screening tests for these clinical populations. Finally, we used multivariable logistic regression with backward elimination (P < .05) to evaluate the combination of best performing screening measures against on-road driving test outcome, adjusting for the data collection site. To examine the predictability of the best performing screening measures combinations, they were further assessed using leave-one-out and 10-fold cross validation. To address missing data in covariates, we carried out sensitivity analysis using a fully-conditional specification²⁹ technique. Twenty imputed data sets were considered, and estimates from these data sets were later combined to obtain the pooled estimates with the Rubin rule. Analyses were conducted using Stata statistical software version 16.0 (StataCorp). P values were 2-sided, and statistical significance was set at P < .05. Data were analyzed from August 1, 2019, to April 2, 2020.

Of 675 individuals who responded to recruitment, 560 participated in the study and also completed the on-road driving test (Figure 1). The age range was 63 to 94 years (mean [SD], 74.7 [6.2] years; 350 [62.5%] men). Participants had a median (interquartile range [IQR]) of 15 (2-12) years of education, and 68 participants (12.1%) were classified as unsafe drivers according to the on-road test. In the Canberra sample, 105 participants were classified as having cognitive impairment (either met MCI criteria or were referred to DARS), and in the Brisbane sample, 124 participants had clinically significant visual impairment. Participants in the comparison group from the Canberra site, compared with those from the Brisbane site, were older (median [IQR] age, 75.0 [71.0 to 80.0] years vs 71.0 [68.0 to 76.0] years), had more years of education (median [IQR], 16.0 [13.0 to 18.0] years vs 14.0 [12.0 to 17.0] years), performed worse on Road Law (median [IQR] score, 32.0 [30.0 to 34.0] vs 35.0 [33.0 to 36.0]), Multi-D (median [IQR] score, −4.44 [−5.27 to −3.47] vs −4.25 [−5.24 to −3.21]), visual acuity (median [IQR] score, 0 [−0.1 to 0.1] vs −0.1 [−0.1 to 0]) and contrast sensitivity (median [IQR] score, 1.7 [1.7 to 1.7] vs 1.8 [1.8 to 1.8]), and on-road test (18 drivers who were unsafe [8.7%] vs 9 drivers who were unsafe [7.0%]), and better on UFOV (median [IQR] score, 83.0 [30.0 to 167.0] vs 97.0 [43.0 to 222.0]), and HPT (median [IQR], 5.5 [4.8 to 6.5] seconds vs 6.1 [5.2 to 7.0] seconds) (Table 1). There were more men in the cognitively impaired (66 [62.9%] men) and visually impaired (87 [70.2%] men) groups than in the comparison groups in Canberra (119 men [58.9%]) and Brisbane (78 [60.5%] men).

In terms of adverse events, at the Canberra site, 1 participant (0.3%) nearly fell during the postural sway test and 3 participants (1.0%) had their on-road tests stopped early owing to safety concerns, but there were no injuries or traffic collisions. At the Brisbane site, 17 participants (6.7%) had their on-road test stopped early, and there were no injuries or traffic collisions.

The 10-point on-road driving test safety rating was dichotomized (unsafe score: ≤3; safe score: >3) and used as the outcome measure in logistic regression models that estimated individual associations of screening measures with driving safety (Table 2). All measures predicted on-road driving test performance in complete case analysis and in the multiply imputed data set in the combined data set of 560 drivers. Sensitivity, specificity, PPV, NPV, and the percentage correctly classified are reported in Table 3. No test had high PPV, but they all had high NPVs (ie, >90.0%), suggesting that these tests correctly classify safe drivers as safe. Overall statistics were best for the Multi-D in these univariate analyses (sensitivity, 77.1%; specificity, 82.1%; PPV, 33.3%; NPV, 96.9%; 81.6% of participants correctly classified).

The dichotomized on-road driving test safety rating was used as the reference standard in ROC analyses for each screening measure. Table 2 displays the AUC for each measure adjusted for site. The AUCs ranged from 0.62 to 0.85 in the complete case analysis, with Multi-D battery test scoring highest (AUC, 0.85 [95% CI, 0.79 to 0.90]) among the measures compared.

Multivariate analyses found that a combination of Multi-D, UFOV, and HPT remained significant when all variables were tested simultaneously in the model in complete case analysis, with an AUC of 0.89 (95% CI, 0.85 to 0.94) (eTable 4 in the Supplement). This model provides an estimated sensitivity of 80.4% and specificity of 84.1%. We obtained a similar AUC from the multiple imputed data sets with same covariates (eTable 4 in the Supplement). Figure 2A shows the ROC curve for the multivariate model, indicating a significant improvement in classification when combining measures. Both 10-fold and leave-one-out cross-validation analyses resulted in very similar AUC estimates for the combination of Multi-D, UFOV, and HPT to the complete case analysis (Figure 2B).

Results for subgroups are shown in eTable 5 in the Supplement, including visually impaired, cognitively impaired, and the comparison group at each site. In the visually impaired group, all measures except the 14-Item Road Law Test and DriveSafe Intersection Test predicted safe driving on the on-road test. In the cognitively impaired group, the maze test, UFOV, and DriveSafe Intersection Test did not identify unsafe drivers. Sensitivity, specificity, PPV, and NPV by subgroup are shown in eTable 6 in the Supplement. The multivariate battery provided the highest sensitivity and specificity in the subgroups (eTable 6 in the Supplement). The multivariate model had 83.3% and 91.8% in the cognitively impaired group and 87.5% sensitivity 70.8% specificity in the visually impaired group.

This prognostic study found that reliable classification of drivers who are unsafe with an off-road screening assessment is possible. Of the individual tests evaluated, the Multi-D, HPT, and UFOV performed well. A combination of these tests provided the most robust assessment of driving safety off-road and could be suitable for clinical groups with visual impairment or cognitive impairment. This is likely to be because a range of measures is needed to identify impairments or weaknesses in the skills and abilities underlying safe driving performance. The Multi-D assessment provided the best prediction of any individual test across subsamples and the combined sample.

A low PPV and high NPV is to be expected for tests for rare conditions, as both statistics are influenced by prevalence. Our 12.1% rate of unsafe driving resulted in a low PPV, although sensitivity and specificity were high. Nevertheless, the rate of unsafe drivers in our sample is comparable to that reported in prior studies that included samples from the general population, such as 5.4% reported by Vaucher et al,³⁰ 16.7% reported by Classen et al,³¹ or 17.4% reported by Wood et al,¹² but lower than the prevalence in samples sourced only through clinics, such as 31.0% reported by Niewoehner et al among veterans referred to a driving clinic¹⁵ and by Kay et al among participants recruited from rehabilitation facilities,¹⁹ and 65.7% reported by Carr et al among drivers with dementia.²² The high sensitivity and specificity for the multivariate model indicate that the PPV would be slightly better than what we observed in our data if the model was used with a sample containing a higher prevalence of drivers who were unsafe.

The results of this study address a critical gap in the evidence on older driver assessment. Our findings demonstrate that it is possible to screen older drivers using off-road tests. This is highly significant because this approach would save the cost and time of an on-road driving assessment for many drivers. With populations aging, many countries lack sufficient resources to provide on-road driving assessments. The use of an objective assessment battery relieves physicians of the need to make subjective or clinical judgements about fitness to drive. We have previously shown that driver screening measures perform better than neuropsychological assessments for identifying unsafe driving.¹³

Further research is required to evaluate the frequency at which older drivers require assessment and to identify cutoffs for safety in specific driving populations. Quantification of crash risk associated with different levels of performance could be possible if large, prospective studies were conducted with these screening tests measured at baseline. Future research could also provide health economic evaluations of screening tools for older drivers.

Our project addressed several limitations of prior studies. To our knowledge, this is the first study to evaluate several off-road screening tests on the same samples, allowing us to compare the utility of each test for identifying drivers who may be unsafe or safe. We included a larger sample with on-road test results than has been studied previously for this purpose, to our knowledge.³² Most prior studies have included fewer than 200 participants, with the previous largest study including 425 participants.³³ Our larger sample provides more precise estimates of the accuracy of these tests in predicting unsafe driving behavior in older adults. Previously, reviews^34,35 have attempted to compare the statistical properties of tests despite tests being evaluated on studies that differ in the prevalence rate of unsafe driving, sample characteristics, and clinical status. In prior research, typically only 1 or 2 tests were evaluated. Our on-road driving test assessors were blind to results of the screening tests, and we evaluated both imputed data sets and complete cases. We also evaluated the screening tests on 2 preclinical groups of drivers who are at highest risk of becoming unsafe on the road and are regularly treated by physicians.

This study also has some limitations, including our use of nonrandom sampling. Participants were recruited via referral and advertisements; therefore, they were not necessarily representative of the older driver population, as they are subject to selection effects. However, this limitation is common to most driving research and also occurs in clinical samples. While on-road tests remain the criterion standard for determining unsafe driving performance, they do not assess performance in all driving situations to which drivers are typically exposed. We did not link our results to state crash records because we have previously shown that these have low accuracy in Australia.³⁶ However further evaluation of these measures in relation to actual crash data, as well as naturalistic driving is warranted.

The findings of this prognostic study suggest that there is sufficient evidence to use off-road screening tests to identify older drivers who may be unsafe. Validated off-road tests are cost effective and efficient, and they could be a more cost-effective approach to manage safety among older drivers.

Accepted for Publication: April 8, 2020.

Published: June 17, 2020. doi:10.1001/jamanetworkopen.2020.8263

Correction: This article was corrected on August 31, 2020, to fix the description in the Methods of the maze test used.

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2020 Anstey KJ et al. JAMA Network Open.

Corresponding Author: Kaarin J. Anstey, PhD, School of Psychology, University of New South Wales, Randwick, Sydney, NSW 2031, Australia (k.anstey@unsw.edu.au).

Author Contributions: Dr Anstey had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Anstey, Eramudugolla, Wood.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Anstey, Eramudugolla, Wood.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Huque, Horswill, Kiely.

Obtained funding: Anstey, Wood.

Administrative, technical, or material support: Anstey, Eramudugolla, Horswill, Black, Wood.

Supervision: Anstey, Eramudugolla, Wood.

Conflict of Interest Disclosures: None reported.

Funding/Support: This study was funded by grants 1045024 and 1102694 from the National Health and Medical Research Council.

Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Additional Contributions: Kerry Mallon, BOcc Thy, and Jasmine Price, BOcc Thy (Neuroscience Research Australia), and Ivor Brooker and Dereck Crook (5th Gear Motoring) conducted on-road driving assessments. All individuals were compensated for their assistance in data collection.