Development and Initial Validation of a Consumer Questionnaire to Predict the Presence of Ear Disease

Key Points

Question  Can a self-administered consumer questionnaire assess an individual’s risk for ear disease?

Findings  In this questionnaire validation study including 307 adults, our prototype instrument was highly sensitive in detecting an individual’s risk for a broad range of ear diseases associated with hearing loss.

Meaning  With an exponential growth of adults with age- and noise-related hearing loss and identified barriers surrounding procurement of hearing aids, initial validation of our questionnaire shows promise for reducing these barriers to care while maintaining public safety through sensitive detection of disease.

Importance  The already large population of individuals with age- or noise-related hearing loss in the United States is increasing, yet hearing aids remain largely inaccessible. The recent decision by the US Food and Drug Administration to not enforce the medical examination prior to hearing aid fitting highlights the need to reengineer consumer protections when increasing accessibility. A self-administered tool to estimate ear disease risk would provide disease surveillance without posing an unreasonable barrier to hearing aid procurement.

Objective  To develop and validate a consumer questionnaire for the self-assessment of risk for ear diseases associated with hearing loss.

Design, Setting, and Participants  The questionnaire was developed using established methods including expert opinion to validate and create questions, and cognitive interviews to ensure that questions were clear to respondents. Exploratory structural equation modeling, logistic regression, and receiver operating characteristic curve analysis were used to determine sensitivity and specificity with blinded neurotologist opinion as the criterion for evaluation. Patients 40 to 80 years old with ear or hearing complaints necessitating a neurotologic examination and a control group of participants with a diagnosis of age- or noise-related hearing loss participated at the Departments of Otorhinolaryngology and Audiology of Mayo Clinic Florida.

Main Outcomes and Measures  Sensitivity and specificity of the prototype questionnaire to identify individuals with targeted diseases.

Results  Of 307 participants (mean [SD] age, 62.9 [9.8] years; 148 [48%] female), 75% (n = 231) were enrolled with targeted disease(s) identified on neurotologic assessment and 25% (n = 76) with age- or noise-related hearing loss. Participants were randomly divided into a training sample (80% [n = 246; 185 with disease, 61 controls]) and a test sample (20% [n = 61; 46 with disease, 15 controls]). Using a simple scoring method, a sensitivity of 90% (95% CI, 84%-94%) and specificity of 72% (95% CI, 59%-82%) were established in the training sample. Applying this cutoff to the test sample resulted in 76% (95% CI, 61%-87%) sensitivity and 80% (95% CI, 51%-95%) specificity.

Conclusions and Relevance  This is the first self-assessment tool designed to assess an individual’s risk for ear disease. Our preliminary results demonstrate a high sensitivity to disease detection. A further validated and refined version of this questionnaire may serve as an efficacious tool for improving access to hearing health care while minimizing the risk for missed ear diseases.

Roughly 30 million Americans have received a diagnosis of hearing loss in both ears.¹ These numbers are growing at exponential rates; 27% of adults 60 to 69 years, 55% of adults 70 to 79 years, and 80% of adults 80 years of age and older have hearing loss.¹ Only a small fraction of these Americans with hearing loss seek and use hearing aids.^2-5 Accessibility and affordability of hearing health care is becoming an issue of national concern, as untreated hearing loss is being associated with a variety of physical, cognitive, social, and economic concerns.^6-8

The President’s Council of Advisors on Science and Technology (PCAST)⁸ and the National Academy of Medicine⁷ (NAM [formerly the Institute of Medicine]) recently submitted reports on hearing loss in the United States, noting associations between untreated hearing loss and communication difficulties, social isolation, depression, dementia, reduced cognition, increased fall risk, and an inability to work, travel, or be physically active.^9-23 Both the PCAST and the NAM reports agreed that untreated hearing loss adversely affects a person’s general physical and cognitive health and reflects a substantial societal health problem. Yet, as the PCAST report states, “While untreated hearing loss likely impairs physical and cognitive health, only a minority of Americans with hearing loss (15-30%) seek out and use assistive hearing technologies.”^8(p1) Even among those with the poorest hearing, only 59% report hearing aid use.² Several barriers to using hearing technologies were identified, including cost and technological shortfalls from limited innovation and market competition. Importantly, they recommended a new hearing aid class that might be purchased “over the counter,” directly by the consumer, with the primary desire of reducing costs.

Both the PCAST and NAM reports were critical of the US Food and Drug Administration’s (FDA’s) regulations surrounding hearing aid procurement, which recommended that all consumers with hearing loss either undergo a medical evaluation to detect ear diseases or sign a waiver before hearing aid purchase. The NAM committee concluded that the FDA policy resulted in little meaningful public health benefit, and recommended the removal of these regulations. Based on these reports, the FDA suspended the enforcement of this policy.²⁴ Those opposed to eliminating the prepurchase medical examination, however, note the potential for serious medical consequences of undetected ear diseases accompanying hearing loss (eg, vestibular schwannoma). In recognition of this surveillance need, the NAM report recommended that over-the-counter amplification devices include clear labeling and information on medical situations, symptoms, or signs for which to consult a physician.

To attain the public health objectives of disease detection while improving accessibility and affordability of hearing health care, we have developed and validated a consumer questionnaire to ascertain risk of ear disease—the Consumer Ear Disease Risk Assessment (CEDRA). The overall goal of CEDRA is to triage consumers into those who likely warrant a medical evaluation prior to hearing aid procurement vs those at lower risk for ear disease. Herein we report the initial results of CEDRA’s ability to screen for ear disease.

A master list of 104 ear diseases collated from standard neurotology texts was rated along multiple dimensions including importance of detection prior to hearing aid fitting by a group of neurotologists, as described in Kleindienst et al.²⁵ A set of questions related to the symptomology of the targeted diseases and the Red Flag questions of the FDA²⁶ and the American Academy of Otorhinolaryngology, Head and Neck Surgery²⁷ were generated by 3 content experts, and formed the initial version of the CEDRA questionnaire.

The prototype CEDRA questionnaire was then refined through an iterative process using cognitive interviews²⁸ with adults eligible for hearing aids (n = 23). Interviewees provided feedback about intelligibility and readability of each question. On the basis of this feedback, CEDRA was modified by eliminating redundant items and reordering items into clusters according to relevance of life events (eg, occupation, major illness). The modified CEDRA was used for a second stage of cognitive interviews (n = 12) and resulted in a version ready for instrument testing after minor modifications.

All study proceedings were approved by the Mayo Clinic Institutional Review Board and included a participant stipend. Convenience sampling was used to recruit patients between the ages of 40 and 80 years seen at the departments of otorhinolaryngology and/or audiology at Mayo Clinic Florida between June 2014 and August 2015 for ear or hearing-related complaints requiring neurotologic examination. Effort was made to recruit participants not previously seen for their complaint when possible. Participants were categorized into targeted ear disease group(s) based on neurotological diagnosis. A control group of participants between the ages of 40 and 80 years with a diagnosis of age- or noise-related hearing loss without targeted ear diseases was also enrolled. Hearing loss sufficient to warrant a recommendation for amplification was not required for participants because in theory, CEDRA would be completed on anyone pursuing treatment for their self-perceived hearing complaint. Total target enrollment for participants with ear disease and controls was 300 based on a power analysis conducted prior to any data collection, which indicated 80% power to detect an area under the curve of 0.63 or more at the 5% level with a sample of that size.

After providing written informed consent, participants completed a paper-and-pencil version of the CEDRA questionnaire with a member of the research group available for assistance. They completed information about their race and ethnicity as per National Institutes of Health criteria. Participants then proceeded to a full audiological and neurotological evaluation as directed by their medical care. The evaluating neurotologist, blind to the participant’s CEDRA responses, assigned a diagnosis or diagnoses based on physical examination results and audiological assessment. If further testing was ordered following initial examination, these reports and any other relevant medical records were reviewed prior to diagnosis. The neurotologist’s diagnosis was the benchmark for evaluating the questionnaire’s ability to detect targeted ear disease.

The primary end points were to establish how well the prototype CEDRA and its constituent items identified participants with a diagnosis of a targeted disease, and to discriminate between participants with a targeted disease and those with age- or noise-related hearing loss.

Data were divided into training and test data sets to first calibrate a model and cutoff score in the training data, and then examine the accuracy of the score and cutoff in the test sample. Before all analyses, we examined the frequency distributions and content of each CEDRA item to identify items that were unlikely to be useful for prediction (eg, items with low variability, redundancy, or zero response rates). We examined eigenvalues from an exploratory factor analysis as a guide to the number of distinct factors that might be present among the CEDRA items. This exploratory factor analysis was used to determine how many factors to include in exploratory structural equation modeling (ESEM), which accomplished 2 important goals in 1 analytic step: (1) group items into cohesive factors and (2) determine the relationship between the factors and the probability of ear disease. We selected ESEM as an analytic technique because some aspects of our model were exploratory (eg, how items grouped together), whereas other aspects of the model were confirmatory (eg, the regression of ear disease on CEDRA). Nearly all variables in the model were specified as categorical. Thus, model estimation was done using the mean- and variance-adjusted weighted least-squares χ² estimator using Mplus software for factor analyses.²⁹ Most CEDRA items were dichotomous except question 27 (Q27), which we defined as a count of the number of possible rare diagnoses and symptoms. Items of Q27 were infrequently or not selected, excluding “Ménière’s disease” and “head trauma” (selected by 11% and 8%, respectively), so these were included as separate variables. Three rating-scale items, numbers 16, 18, and 19, were analyzed as a dichotomy (eg, “How often do you have dizziness?” was scored as 0 for “never” and “occasionally” and as 1 for “sometimes” and “always”). Factors in the ESEM were subjected to oblique GEOMIN rotation to enhance interpretability.²⁹ Based on the ESEM, we selected CEDRA items for inclusion in a simple scoring algorithm. We then tested whether these items, using a summed score, could predict disease status using logistic regression. A cutoff score was selected using receiver operating characteristic curve analysis and the predicted probability of disease as a function of CEDRA score. This cutoff score was then applied to the test sample to calculate sensitivity and specificity. The test sample data were not used in the creation of the algorithm or the selection of the cutoff.

Three hundred seven participants (mean [SD] age, 62.9 [9.8] years; 148 [48%] female) were included in the analyses. Table 1 presents a summary of demographic data. Of the 307 participants, 75% (n = 231) were enrolled with targeted disease(s) and 25% (n = 76) were enrolled in the control group with diagnosed age- or noise-related hearing loss. For analyses, all targeted diseases were collapsed into 1 dichotomous absence/presence variable.

The 307 participants were randomly divided into 2 groups, a training sample (80% [n = 246; 185 with disease, 61 controls]) and a test sample (20% [n = 61; 46 with disease, 15 controls]). The eTable in the Supplement presents a breakdown of disease type and controls by sample. An 80/20 split was used to ensure a large enough training sample to build a robust model. A post hoc power analysis with the software program G*Power³⁰ indicated that statistical power was 87% to detect an odds ratio of 1.50 or greater for a binary (50/50) outcome.

In keeping with our goal of finding a simple solution, we began by specifying 3 factors (eigenvalues >4.0) from the exploratory factor analysis. The first 3 eigenvalues (11.93, 6.17, and 4.70) were relatively large compared with the others, although the scree plot (Figure 1) lends itself to multiple interpretations (eg, the large first eigenvalue might suggest a unidimensional or bifactor model, whereas as many as 5 factors might be present based on the knee). The more complex 5-factor solution yielded similar results to the simpler 3-factor model and is not presented here because our goal was to optimize the prediction of disease as simply as possible and the exact number of factors was not critical to forming the algorithm. The final ESEM model was used to create a simple, 1-score solution for CEDRA.

Table 2 provides the numerical results for the 3-factor ESEM model (Figure 1). Expectedly, factor 1 was not predictive of ear disease because it mostly included questions related to noise exposure, tinnitus, and head trauma. Factors 2 and 3 included items mostly related to disease history and ear or hearing symptoms. They were similarly predictive of the presence of disease, with regression coefficients of 0.53 and 0.59, respectively. Approximately 30% of the items in factors 2 and 3 loaded onto both factors with factor loadings of 0.4 or greater. Based on the ESEM, we created a simple scoring algorithm that added up the items with loadings of at least 0.4 from either factor 2 or 3 (see eResults in the Supplement for the list of items). Only items that loaded onto 1 factor were included to reduce redundancy. After selecting these items, we used logistic regression to determine the probability of any targeted disease as a function of this simple score using the training sample. There was a significant association between CEDRA score and presence of disease (odds ratio, 1.84; 95% CI, 1.56-2.17).

A cutoff score of at least 4 on the questionnaire was chosen as high risk for a targeted ear disease. This cutoff was chosen on the basis of predicted probabilities from the logistic regression analysis. A predicted probability of 0.64 or greater was achieved with a CEDRA score of 4 or greater (ie, more likely to have a targeted disease) and resulted in a sensitivity of 90% (95% CI, 84%-94%) and specificity of 72% (95% CI, 59%-82%) in the training sample (Figure 2 shows a plot of the receiver operating characteristic curve). Applying this cutoff to the test sample (n = 61) resulted in 76% (95% CI, 61%-87%) sensitivity and 80% (95% CI, 51%-95%) specificity. Results for the test sample were considered acceptable as training group data were included in model optimization, parameter estimation, and cutoff selection.

Access to hearing health care and amplification devices is a public health issue of increasing importance. Previous FDA regulations regarding procurement of amplification devices, developed in the 1970s, emphasized the need for a prepurchase medical evaluation, preferably by physicians specializing in diseases of the ear; however, based on the recommendations from the PCAST and NAM, the FDA has decided not to enforce the requirement for physician examination or medical waiver prior to hearing aid purchase to improve accessibility.²⁴ Despite elimination of this barrier to hearing health care, there is concern for medical consequences of ear diseases accompanying hearing loss if left undetected. There is a need to find a method that maintains public safety while increasing accessibility to hearing health care.

The detection model involving a medical examination is costly because the majority of hearing loss in the adult US population is age or noise related.⁷ This is further exacerbated by physician shortage³¹ and the clustering of specialty physicians, such as otolaryngologists specializing in ear disease, in major metropolitan areas,³² making specialist examination prior to hearing aid procurement for rural populations even more difficult.

Our innovative questionnaire for monitoring diseases associated with hearing loss is a potential solution for communities without access to specialty physicians. Our tool was 90% sensitive and 72% specific in a training sample of 246 individuals. It was 76% sensitive and 80% specific in a test sample (n = 61). Despite these smaller values, which can occur when data are not “fit” to the sample, our questionnaire shows early promise as a tool for consumers with hearing difficulty to self-screen for disease prior to procuring hearing aids. Taken in context, in its current form CEDRA would miss 24% of individuals with disease and would overrefer 20% based on the test sample. Because of the decision of the FDA not to enforce the medical examination, CEDRA theoretically increases public safety by 76% without increasing the cost associated with medical examination. However, it would also refer 20% of individuals for an examination who would not benefit from it. Although CEDRA in its preliminary state appears sensitive to disease, further study and refinement is necessary to improve its accuracy.

We developed a simple scoring algorithm to determine whether an individual is at risk for disease based on their CEDRA responses using exploratory structural equation modeling. Although there are many approaches to scoring a questionnaire, we found that a simple sum score was effective. We explored multiple-factor solutions, as well as more complex algorithms (eg, differentially weighting items and/or subscales), but none outperformed the simple scoring algorithm that we used, which essentially summed CEDRA items.

Developing this tool comes at a time of change in hearing health care. The PCAST and NAM have identified affordability and accessibility of hearing aids for adults as substantial hurdles in hearing health care. Both reports encourage a direct-to-consumer approach for the purchase of hearing aids and other amplification devices. In parallel, market pressures are creating an increase in the availability of over-the-counter hearing aids and hearing aid–related technology³³ on the internet and at large-scale retailers. Likewise, the development and sale of consumer electronic devices such as personal sound amplifier products and “hearables” are expanding rapidly.

Our work provides encouraging preliminary evidence that consumer-based disease surveillance prior to hearing aid purchase is feasible. Our CEDRA questionnaire demonstrated high sensitivity to the detection of disease, which is important because of the potential consequences of many of the diseases associated with hearing loss. Encouragingly, CEDRA, even in its preliminary version, appears to be sensitive to a broad range of low-prevalence conditions. These findings, however, are based on convenience sampling and an artificially high incidence of targeted diseases, resulting in selection bias and potentially exaggerated estimates of diagnostic accuracy. Furthermore, not all participants in this sample were unaware of their diagnosis prior to completing CEDRA, potentially biasing their responses. A larger-scale effort on a more representative population of naive hearing aid seekers is under way to estimate the broader performance of CEDRA and to ensure robust, replicable, and unbiased results. A Spanish version of CEDRA was also developed and is being evaluated.

Hearing health care is lacking an economic surveillance tool for ear disease. Our consumer-focused questionnaire, once validated on a larger sample, would provide such a standardized tool of disease screening and would mitigate safety concerns across all avenues of amplification device distribution. CEDRA not only satisfies the NAM recommendations for increased information dissemination to consumers about safety related to amplification use, it expands on this goal by providing actionable information to the consumer in the form of a risk score.

CEDRA represents a unique and important preliminary tool to enhance and economize public safety in hearing health care by using a consumer questionnaire to assess an individual’s risk for ear disease. CEDRA comes at a pivotal time, is responsive to the recommendations of PCAST and NAM, and provides public policymakers and stakeholders with new preliminary information on which to base decisions for the future of hearing health care.

Accepted for Publication: May 20, 2017.

Correction: This article was corrected on March 8, 2018, to fix errors in sensitivity and specificity, odds ratios, and 95% CIs caused by an error in the analysis code for 3 of the questionnaire items; also, Figure 2 has been replaced and the figure caption updated.

Corresponding Author: Samantha J. Kleindienst, PhD, Department of Audiology, Norton Sound Health Corporation, Nome, AK 99762 (skleindienst@nshcorp.org).

Published Online: August 3, 2017. doi:10.1001/jamaoto.2017.1175

Author Contributions: Drs Zapala and Dhar had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Kleindienst, Zapala, Nielsen, Griffith, Lundy, Dhar.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Kleindienst, Zapala, Nielsen, Griffith, Lundy, Dhar.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Kleindienst, Zapala, Griffith.

Obtained funding: Zapala, Nielsen, Dhar.

Administrative, technical, or material support: Kleindienst, Zapala, Nielsen, Rishiq, Dhar.

Supervision: Zapala, Nielsen, Lundy, Dhar.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported.

Funding/Support: This work was supported by grants from the National Institutes of Health/National Institute on Deafness and Other Communication Disorders (R21/33 DC013115 to Drs Dhar and Zapala), the Knowles Hearing Center at Northwestern University (to Drs Dhar, Zapala, and Nielsen), and by the James Russell and Martha Crawford Endowed Clinical Research Fellowship in Otolaryngology (to Drs Kleindienst and Rishiq).

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Additional Contributions: The authors thank Deborah L. Carlson, PhD, University of Texas Medical Branch, Galveston, for her close editorial review. She received no compensation for her contribution.