When presented with an audio track playing /pa/ and a separate video track of a person silently articulating /ka/, participants will most often claim to hear a unique fusion sound /ta/.
The colored bars represent the duration between the diagnosed onset (labeled “O”) of their amblyopia to resolution (labeled “R”), last examination showing unresolved amblyopia (labeled “U”), and visual auditory-integration testing date (labeled “T”). Periods of unknown definitive amblyopia onset are labeled “O?” and the blank segments indicate periods of resolved amblyopia prior to testing. The green bars indicate that the child perceived the McGurk effect, whereas the red bars indicate failure to perceive the McGurk effect. The children (ie, patients) are separated into groups based on the onset and resolution dates.
When presented with an audio track playing /pa/ and a separate video track of a person silently articulating /ka/, viewers will most often claim to hear a unique fusion sound /ta/ as shown in this video.
Burgmeier R, Desai RU, Farner KC, et al. The effect of amblyopia on visual-auditory speech perception: why mothers may say “look at me when I’m talking to you.” JAMA Ophthalmol. Published online September 11, 2014. doi:10.1001/jamaophthalmol.2014.3307.
Burgmeier R, Desai RU, Farner KC, Tiano B, Lacey R, Volpe NJ, Mets MB. The Effect of Amblyopia on Visual-Auditory Speech PerceptionWhy Mothers May Say “Look At Me When I’m Talking To You”. JAMA Ophthalmol. 2015;133(1):11-16. doi:10.1001/jamaophthalmol.2014.3307
Children with a history of amblyopia, even if resolved, exhibit impaired visual-auditory integration and perceive speech differently.
To determine whether a history of amblyopia is associated with abnormal visual-auditory speech integration.
Design, Setting, and Participants
Retrospective observational study at an academic pediatric ophthalmologic clinic with an average of 4 years of follow-up. Participants were at least 3 years of age and without any history of neurologic or hearing disorders. Of 39 children originally in our study, 6 refused to participate. The remaining 33 participants completed the study. Twenty-four participants (mean [SD] age, 7.0 [1.5] years) had a history of amblyopia in 1 eye, with a visual acuity of at least 20/20 in the nonamblyopic eye. Nine controls (mean [SD] age, 8.0 [3.4] years) were recruited from referrals for visually insignificant etiologies or through preschool-screening eye examinations; all had 20/20 in both eyes.
Participants were presented with a video demonstrating the McGurk effect (ie, a stimulus presenting an audio track playing the sound /pa/ and a separate video track of a person articulating /ka/). Normal visual-auditory integration produces the perception of hearing a fusion sound /ta/. Participants were asked to report which sound was perceived, /ka/, /pa/, or /ta/.
Main Outcome and Measure
Prevalence of perception of the fusion /ta/ sound. Prior to the study, amblyopic children were hypothesized to less frequently perceive /ta/.
The McGurk effect was perceived by 11 of the 24 participants with amblyopia (45.8%) and all 9 controls (100%) (adjusted odds ratio, 22.3 [95% CI, 1.2-426.0]; P = .005). The McGurk effect was perceived by 100% of participants with amblyopia that was resolved by 5 years of age and by 100% of participants whose onset at amblyopia developed at or after 5 years of age. However, only 18.8% of participants with amblyopia that was unresolved by 5 years of age (n = 16) perceived the McGurk effect (adjusted odds ratio, 27.0 [95% CI, 1.1-654.0]; P = .02).
Conclusions and Relevance
This pilot study suggests that children with a history of amblyopia have impaired visual-auditory speech perception. Early childhood appears to serve as an approximate time point for the development of successful visual-auditory fusion, by which time amblyopia must have either resolved or begun. Interventions to resolve amblyopia may not only influence visual acuity but may also influence the perception of sound.
Children do not just hear with their ears; they also “hear” with their eyes. Speech perception is a multimodal phenomenon, integrating auditory input with the visual input of the speaker’s oral and extraoral facial movement.1- 3 One of the most striking demonstrations of how vision influences the perception of sound is the McGurk effect.4 In our demonstration, when presented with an audio track playing /pa/ and a separate video track of a person silently articulating /ka/, participants will most often claim to hear a unique fusion sound /ta/ (Figure 1; Video).
The McGurk effect has served as a well-studied demonstration of the normal process of visual-auditory integration, with approximately 80% to 95% of individuals perceiving the fusion sound.5- 8 However, because no published study to our knowledge has assessed visual acuity monocularly among test participants, it is unclear whether amblyopia is a confounding factor in speech perception. The purpose of our study is to determine whether a history of amblyopia affects visual-auditory integration. We propose that a critical time period exists during early childhood when amblyopia may resolve or occur to influence visual-auditory integration and demonstrate that binocularity affects the perception of sound in children with amblyopia.
Our study was approved by the Ann and Robert H. Lurie Children’s Hospital of Chicago institutional review board. All participants provided assent and all of their parents provided written informed consent. None of the participants or their parents were financially compensated. All ophthalmology examinations and interventions were performed by board-certified pediatric ophthalmologists. A thorough review of medical records was performed for all participants prior to patient selection.
The inclusion criteria for controls were that they had to be at least 3 years of age (ie, earliest age to reliably understand the instructions), have 20/20 visual acuity in each eye, and have 40 arcseconds of stereoacuity. Controls were recruited from children referred for visually insignificant periocular etiologies (eg, chalazia) or through preschool-screening eye examinations.
A nonconsecutive series of patients with amblyopia were recruited for our study. The inclusion criteria for these patients were that they had to be at least 3 years of age, have received a diagnosis of amblyopia (regardless of history of treatment or resolution), and have at least 20/20 visual acuity in the nonamblyopic eye. Age at onset of amblyopia was determined by a medical record review of the earliest signs, parental history, or a referring primary care professional’s detection of an abnormal red reflex or an abnormal Hirschberg test result, and a difference in visual acuity between eyes of at least 2 Snellen equivalent lines. Successful treatment of amblyopia was considered to have occurred when the patient’s visual acuity was equal in both eyes and when patching was no longer required.
The inclusion criteria for all participants also included use of oral language as a primary form of communication in their primary language, not limited to English speakers. The exclusion criteria for all participants were as follows: any history of developmental delay, neurologic disorders (eg, seizure, brain injury, cerebrovascular accident, congenital malformation, neoplasm, or autism spectrum disorder), or hearing disorders.
Visual acuity was assessed at a distance of 6 m (20 ft) by crowded-bar individual Allen symbols or crowded-bar individual Snellen letters, depending on the age of the patient. All reported visual acuities were best-corrected visual acuities, with the exception that participants who had a visual acuity of 20/20 or better in both eyes were not refracted. Stereoacuity was assessed with the Stereo Fly SO-001 test (Stereo Optical). Alignment was assessed with alternate prism-cover testing.
Stimuli were presented with a QuickTime video on Microsoft PowerPoint using Mac OS X 10.6 (Apple Inc) (Figure 1). The video images measured 25 cm (10 in) diagonally, centered on a 38-cm (15-in) MacBook Pro laptop monitor (Apple Inc) with a black backdrop. Participants were seated 36 cm (14 in) away from the monitor, facing the screen at 0° azimuth. Laptop speakers, set to deliver 75-dB volume at the participant’s distance, were used to deliver the audio stimuli.
Seven types of stimuli were displayed: unimodal audio-only /pa/ (ie, with a blank screen), audio-only /ka/, visual-only /pa/ (ie, with no sound), visual-only /ka/, bimodal congruent pair of audio /pa/ with visual /pa/, bimodal congruent pair of audio /ka/ with visual /ka/, or the McGurk effect stimulus of an incongruent visual /ka/ dubbed with an audio /pa/ (Video). Each stimulus was presented as a series of 3 repetitions over 5 seconds.
The participants were tested in 2 stages. In both stages, amblyopic children were recruited regardless of age at onset or resolution of amblyopia. The controls were tested during the first stage. In the first stage, 10 trials of each of the 7 stimuli were presented in a random order. The participants (15 amblyopes and 9 controls) were instructed to report if they heard /pa/, /ka/, or /ta/, and their answers were recorded. The research associate recording the answers was masked to both the stimulus presented and the expected response. In the first stage, all participants only viewed the stimuli binocularly. In the second stage, the participants (9 amblyopes) were tested binocularly and monocularly (ie, occluding the fellow eye with an opaque patch). To reduce total length of testing, participants viewed 10 McGurk effect stimuli and 3 of each of the other 6 stimuli. Eye dominance was assessed by asking the participants to hold their hands together in front of them to form a circle, viewing an object at a distance and slowly bringing their hands closer to their face. The eye that was used to fixate on the object was identified as the dominant eye.
Participants who did not perceive the audio-only stimuli at least 66% of the time were referred for formal audiology testing and were excluded from analysis. Participants were also excluded if they did not correctly identify the visual-only /pa/ stimulus or the congruent bimodal stimuli (ie, visual-auditory /pa/pa/ or /ka/ka/), or if they appeared to be inattentive or uncooperative during testing. The participants who perceived the McGurk effect (ie, visual /ka/ with audio /pa/) in at least 80% of the trials were considered to have normal visual-auditory integrative speech perception, whereas the participants who perceived the effect in less than 20% of the trials were considered to have abnormal visual-auditory integrative speech perception.
Statistical analysis was performed using Microsoft Excel 2013. Age was rounded to the nearest year. Snellen equivalent visual acuity was converted to logMAR equivalents for analysis and then reconverted back to Snellen equivalent visual acuity for purposes of presentation. Groups were compared using the Fisher exact test, the t test, or χ2 analysis.
The mean (SD) age was 7.0 (1.5) years (range, 3-9 years) for the 24 participants with amblyopia and 8.0 (3.4) years (range, 4-14 years) for the 9 controls (P = .30).
The McGurk effect was perceived by all 9 controls (100%) but by 11 of 24 participants with amblyopia (45.8%) (adjusted odds ratio, 22.3 [95%, CI, 1.2-426.0]; P = .005). An association could not be identified between the visual-auditory fusion and the etiology of amblyopia, whether through anisometropia (2 participants), strabismus (13 participants), or deprivation (9 participants) (P = .23). No association could be identified between visual acuity or stereopsis and visual-auditory integration.
Among the 24 amblyopic patients who viewed the stimulus binocularly, the mean visual acuity in the amblyopic eye was 20/51 for those who perceived the McGurk effect and 20/32 for those who did not perceive the effect (P = .23). Among the 9 patients tested monocularly, when viewing the stimulus with only the amblyopic eye, the mean visual acuity was 20/74 for those who perceived the McGurk effect (n = 7) and 20/50 for those who did not (n = 2) (P = .47). Participants who failed to perceive the McGurk effect had a mean (SD) stereopsis of 1692 (1506) arcseconds, and those who perceived the McGurk effect had a mean (SD) stereopsis of 875 (1225) arcseconds (P = .17). When visual-auditory fusion was analyzed by stereopsis ranges of 0, less than 1000, and greater than 1000 arcseconds, the association was not statistically significant (P = .21) (Table 1).
For the purpose of analysis, the participants were retrospectively divided into 2 groups: children whose onset of amblyopia was prior to 5 years of age (group 1) and children whose onset of amblyopia was on or after 5 years of age (group 2). Group 1 was subdivided into 2 subgroups: those whose amblyopia resolved before 5 years of age (group 1A) and those whose amblyopia was unresolved at 5 years of age (group 1B) (Figure 2 and Table 2).
Among the 5 participants in group 2, the etiology of amblyopia was mild anterior polar or posterior subcapsular cataracts in 3 participants, a corneal laceration in 1 participant, and accommodative esotropia in 1 participant. A representative participant of group 2, patient GU, received a diagnosis of juvenile dermatomyositis at 3 years of age, was treated with oral prednisone and was examined by his primary care provider several times a year. An abnormal red reflex was first documented at 5 years of age and a follow-up ophthalmologic examination revealed a visual acuity of 20/20 in the right eye and 20/40 in the left eye, with a small posterior subcapsular cataract and amblyopia in the left eye. After a regimen of patching, the participant’s visual acuity improved to 20/20 in both eyes with 40 arcseconds of stereopsis by 7 years of age. Visual-auditory integration was normal for this participant. In all 5 cases of late-onset amblyopia, intact visual-auditory integration was similarly present.
In contrast, among the 19 participants in group 1 (in both subgroups 1A and 1B), 6 (31.6%) perceived the McGurk effect, (adjusted odds ratio, 22.8 [95% CI, 1.1-478.8]; P = .01). All 3 participants in subgroup 1A perceived the McGurk effect, whereas only 3 of the 16 participants in subgroup 1B (18.8%) perceived the McGurk effect (adjusted odds ratio, 27.0 [95% CI, 1.1-654.0]; P = .02). As a representative participant of subgroup 1A, patient OT initially presented with esotropia in the left eye at 3 months of age. Having undergone bilateral medial rectus recession within a few weeks of symptom onset, with subsequent patching, this patient’s amblyopia had resolved by 4 years of age, with a visual acuity of 20/20 in both eyes. As with other cases of early-onset amblyopia resolved by 5 years of age, visual-auditory integration remained intact for all 3 participants in subgroup 1A. In contrast, as a representative of subgroup 1B, patient KM initially presented with esotropia and hypertropia in the left eye at 2 months of age and underwent a bilateral medial rectus recession at 1 year of age, with subsequent superior rectus recession of the left eye at 3 years of age. The earliest visual acuity reported preoperatively at 3 years of age was 20/25 in the right eye and 20/60 in the left eye. In spite of adherence to patching, the results of biannual visits up to 7 years of age have not yet demonstrated resolution of amblyopia, with the most recent visual acuity reported to be 20/20 in the right eye and 20/60 in the left eye. Patient KM did not perceive the McGurk effect.
Our pilot study suggests that children with a history of amblyopia have impaired visual-auditory speech integration. Early childhood appears to serve as an approximate point for the development of successful visual-auditory fusion, by which time amblyopia must have either resolved or begun. However, no association could be identified between visual acuity or stereoacuity and the effect of amblyopia on speech perception.
Early-onset amblyopia that did not resolve by 5 years of age (ie, subgroup 1B) was generally associated with failure of visual-auditory fusion (Figure 2 and Table 2). This was demonstrated by patient KM, who had unresolved amblyopia up to 7 years of age and could not perceive the McGurk effect.
While early-onset amblyopia that was unresolved by 5 years of age (subgroup 1B) was associated with poor visual-auditory integration, early-onset amblyopia that was resolved by 5 years of age (subgroup 1A) was associated with intact integration. As an example of this association, patient OT had amblyopia that resolved by 4 years of age and intact visual-auditory integration. Hence, for early-onset amblyopic participants, our results suggest that early childhood may serve as an approximate point by which amblyopia must have resolved for a patient to maintain normal speech perception.
In contrast, when the onset of amblyopia occurs relatively late in childhood (which we define as ≥5 years of age [group 2]), deficits in speech perception may be spared. Patient GU, a representative participant in group 2 with onset of amblyopia at 5 years of age that improved with patching, had normal visual-auditory integration.
Our hypothesis for a possible point within early childhood for which the resolution of amblyopia is required to maintain visual-auditory fusion is consistent with a study9 of children with presumably intact vision who were born deaf. Among children who would later receive cochlear implants, the likelihood of consistent perception of the McGurk effect decreased with age of implant beyond 2.5 years. Other studies5,10- 12 have suggested that imprinting of the McGurk effect may begin as early as in infancy.
The relatively small sample size is one of several limitations of our study. Although our pilot study finds statistically significant results for age at onset and age at resolution, it was not powered to detect subgroup analysis for mechanisms of amblyopia, stereopsis, or visual acuity.
Another limitation of our study is the lack of required formal audiology testing. However, our study was designed to refer participants for formal audiology testing if they failed to correctly perceive the audio-only stimuli /pa/ sound or /ka/ sound (ie, with visual tract as the blank screen) in less than 66% of trials, signaling a concern for hearing impairment. No child required such further audiology testing. In addition, no standardized tests for intelligence or grammatical understanding were performed, even though children were excluded if their parents suspected or if their pediatricians diagnosed any neurological pathology apart from amblyopia, including developmental delay, attention-deficit/hyperactivity disorder, or dyslexia.
Language learning impairment itself may confound our results because it has been reported to result in the individual’s failure to perceive the McGurk effect. Although all the participants in a study by Norrix et al13 passed an audiology screening test, none of them were assessed for visual acuity. In another study by Hayes et al,14 children with learning disabilities compared with normal-learning children failed to perceive the McGurk effect during testing with ambient noise. Although both groups of participants passed binaural hearing thresholds, the participants’ visual acuities were only assessed binocularly, and hence monocular amblyopic patients may have been included among the study participants, confounding the results. Furthermore, other studies15- 19 have demonstrated that children whose limited language development is associated with autism spectrum disorder often demonstrate poor face-to-face communication and generally fail to perceive the McGurk effect. The possible contributing effects of auditory or visual impairment were not clear because these studies15- 19 did not assess the children using either a hearing or vision test. Our study contributes to the literature by demonstrating how amblyopia itself can influence the perception of the McGurk effect, in addition to the influences of other factors such as hearing, language, and intelligence, which we did not specifically test.
Although our data analysis suggests that 5 years of age may potentially approximate the point for visual-auditory fusion, we urge significant caution with regard to identifying a specific age as a cutoff, owing to several further limitations of our study. First, because of the retrospective nature of our study, the onset of amblyopia was not determined by prospective ophthalmology examinations or Teller Acuity Chart testing since birth, but rather through review of signs suggestive of ocular pathology from parents’ histories and chart reviews of referring primary care providers. Because ophthalmology examinations were scheduled based on clinical need, several months, or occasionally years, passed between examinations. Recall bias, therefore, is inherent in the methods. Moreover, most patients were unable to reliably communicate visual acuity through Allen or Snellen charts until 3 to 5 years of age, possibly delaying our diagnosis of amblyopia from its true onset. Furthermore, because many of our amblyopic patients were initially referred through vision-screening programs that occurred between 4 and 6 years of age, a diagnosis of amblyopia may have occurred sooner had a comprehensive eye examination been performed earlier. Nevertheless, because the majority of such referred patients were within 4 to 6 years of age, we arbitrarily chose 5 years as the dichotomous variable for analysis. Finally, because age was rounded to the nearest year, our results should not be interpreted as assuming that several months are suggestive of any clinical consequence.
Although our pilot study presents new evidence suggesting how amblyopia affects the perception of sound, larger prospective clinical studies are required to further elucidate the potential downstream consequences in higher cortical language development. Future areas of research include testing amblyopic patients for associations with other groups of participants who also fail to perceive the McGurk effect (those with functional magnetic resonance imaging defects in the left superior temporal sulcus, autism spectrum disorders, and language learning disorders).15- 19
Submitted for Publication: March 6, 2014; final revision received May 21, 2014; accepted July 14, 2014.
Corresponding Author: Rajen U. Desai, MD, Department of Ophthalmology, Feinberg School of Medicine, Northwestern University, 645 N Michigan Ave, Ste 440, Chicago, IL 60611 (firstname.lastname@example.org).
Published Online: September 11, 2014. doi:10.1001/jamaophthalmol.2014.3307.
Author Contributions: Dr Desai had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Burgmeier, Desai, Volpe.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Burgmeier, Desai, Farner, Tiano, Volpe.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Burgmeier, Tiano.
Obtained funding: Burgmeier, Desai, Tiano.
Administrative, technical, or material support: Burgmeier, Desai, Tiano, Lacey, Volpe.
Study supervision: Burgmeier, Desai, Volpe, Mets.
Conflict of Interest Disclosures: None reported.
Funding/Support: Dr Desai is supported by the Heed Ophthalmic Foundation and the Society of Heed Fellows. The design and conduct of this study was supported by the grants from the Knights Templar Eye Foundation Inc and Research to Prevent Blindness.
Role of the Sponsor/Funder: The funding agencies had no role in the design and conduct of the study; collection, management, analysis, or interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Additional Contributions: We thank Eric I. Knudsen, PhD, at the Stanford School of Medicine, Stanford, California, for study concept and Seth Greenwood, MS, at the Harvard School of Public Health, Boston, Massachusetts, for statistical analysis. Neither Dr Knudsen nor Mr Greenwood received any financial compensation for their contributions.