Visual Fixation Patterns During Viewing of Naturalistic Social Situations as Predictors of Social Competence in Individuals With Autism

Background  Manifestations of core social deficits in autism are more pronounced in everyday settings than in explicit experimental tasks. To bring experimental measures in line with clinical observation, we report a novel method of quantifying atypical strategies of social monitoring in a setting that simulates the demands of daily experience. Enhanced ecological validity was intended to maximize between-group effect sizes and assess the predictive utility of experimental variables relative to outcome measures of social competence.

Methods  While viewing social scenes, eye-tracking technology measured visual fixations in 15 cognitively able males with autism and 15 age-, sex-, and verbal IQ–matched control subjects. We reliably coded fixations on 4 regions: mouth, eyes, body, and objects. Statistical analyses compared fixation time on regions of interest between groups and correlation of fixation time with outcome measures of social competence (ie, standardized measures of daily social adjustment and degree of autistic social symptoms).

Results  Significant between-group differences were obtained for all 4 regions. The best predictor of autism was reduced eye region fixation time. Fixation on mouths and objects was significantly correlated with social functioning: increased focus on mouths predicted improved social adjustment and less autistic social impairment, whereas more time on objects predicted the opposite relationship.

Conclusions  When viewing naturalistic social situations, individuals with autism demonstrate abnormal patterns of social visual pursuit consistent with reduced salience of eyes and increased salience of mouths, bodies, and objects. Fixation times on mouths and objects but not on eyes are strong predictors of degree of social competence.

ADVANCES IN psychological research of the core social deficits in autism have increasingly focused on disruptions in early-emerging skills that seem to derail the processes of socialization.¹ Typically developing infants show preferential attention to social rather than inanimate stimuli,² and they also prefer to focus on the more socially revealing features of the face, such as the eyes rather than the mouth³; in contrast, individuals with autism seem to lack these early social predispositions.^4-8 This attenuation to the social world is accompanied by documented abnormalities in face perception, both in face recognition and in identification of facial expressions.^9-12 In addition, individuals with autism use atypical strategies when performing such tasks, relying on individual pieces of the face rather than on the overall configuration.^5,8 Alongside these perceptual anomalies, individuals with autism have deficits in conceiving other people's mental states (in having a "theory" that other people have minds and then using this knowledge to predict their social behavior).¹³

Although these findings offer specific hypotheses for the profound impairment in social adjustment exhibited by individuals with autism, there is no performance-based method for directly quantifying the impact of these underlying processes on social functioning in everyday situations. For an individual with autism, the real world is fraught with social challenges and ambiguities that are largely minimized within the narrow parameters of experimental situations. Individuals with autism encounter greater difficulty when trying to spontaneously impose social meaning onto what they see than they do when solving explicit tasks.¹⁴ This is particularly evident in cognitively able individuals, who may use their visual-perceptual or language abilities to scaffold their performance, thus achieving relatively high scores on a given task but doing so in ways that differ from normative strategies.¹⁵ This hypothesis was recently substantiated in a neurofunctional study¹⁶ of face perception in autism in which adequate task performance was accompanied by abnormal ventral temporal cortical activity, which in turn suggested that participants had "treated" faces as objects.

The development of more naturalistic paradigms that capitalize on the challenging nature of open-ended social scenarios for individuals with autism could result in several advantages. First, the effect size of between-group findings may increase, bringing experimentally measured abnormalities in autism to a level more commensurate with clinical observations of social impairment. Second, by placing a performance-based method into the context of naturalistic social functioning, there is increased likelihood that the experimental method will better predict level of social competence. Abnormal performance on social tasks in autism is typically reported as a group finding relative to a control sample, with little attempt to use the given measure as a predictor of level of social adjustment or of level of social impairment. The lack of quantifiable indices of social competence that could define a spectrum of social outcomes, from normality to varying manifestations of autism, hinders genetic and neurofunctional research, which partially rely on such indices for interpretation of heritability and neuroimaging data.^17,18

To create an experimental paradigm to measure social functioning in a context that more closely resembles social demands in naturalistic situations, we used eye-tracking technology to study spontaneous viewing patterns of cognitively able individuals with autism and age- and verbal IQ–matched control subjects. Previous work^19,20 has used static images to assess aspects of face scanning; to our knowledge, this is the first study of eye tracking as a dynamic phenomenon. The paradigm involves viewing digitized videotape clips of complex social situations while wearing a noninvasive eye-tracking device that superimposes the viewer's point of regard onto the viewed scenes. The resultant videotape can then be coded for patterns of visual fixation that can be measured in terms of percentage of viewing time spent on different aspects of the social scenes. For this exploratory study, several lines of research guided our decision as to the scheme to use for coding fixations. First, we were interested in the relative salience of major components of the viewed scenes.^4,8,14,15 Thus, we divided the total on-screen area into face, body, and object regions. Second, given the research results suggesting that individuals with autism have difficulty interpreting social information conveyed through the eyes,^6,7 and that they focus preferentially on mouths when performing face perception tasks,⁵ we subdivided the face into an eye region and a mouth region. The resultant coding scheme, therefore, consisted of 4 clearly delineated viewing areas of interest: mouth, eye, body, and objects (see the "Coding Procedures" subsection for further details). We were also interested in exploring the relationship between patterns of visual fixation and outcome measures of social competence. We operationalized social competence in terms of level of real-life social adjustment (as defined by scores on the socialization domain of the Vineland Adaptive Behavior Scales, Expanded Edition [VABS-E]²¹)and in terms of degree of autistic social symptoms (as defined by the social domain of the Autism Diagnostic Observation Schedule [ADOS]²²). Whereas the VABS-E provides a measure of social ability (higher scores mean higher levels of social adjustment), the ADOS provides a measure of social impairment (higher scores mean higher levels of autistic social behaviors).

On the basis of the extant literature, we expected that individuals with autism would preferentially focus on the mouth region rather than on the eye region and on objects relative to controls. In addition, given the normative pattern of preferentially using information gathered from the eyes to understand others, we also predicted that within the autistic group, higher percentage of viewing time on the eye region would be positively correlated with level of social adjustment and negatively correlated with level of social impairment.

Fifteen male adolescents and young adults with autism were recruited through a large, federally funded research project on the neurobiology of autism carried out in the developmental disabilities section of the Yale Child Study Center. This project includes a 3-day protocol consisting of extensive diagnostic, neuropsychological, neuroimaging, and genetic studies. Before participation, all individuals or their legal guardians supplied written informed consent. The protocol was approved by the Human Investigations Committee of the Yale University School of Medicine. Diagnoses were assigned on the basis of parental interview (Autism Diagnostic Interview–Revised²³)and direct observations of participants' social and communicative behaviors(ADOS²²). All participants with autism met DSM-IV²⁴ diagnostic criteria for autistic disorder as operationalized through standardized algorithms derived from the Autism Diagnostic Interview–Revised and the ADOS. Intellectual level was measured using the Wechsler Intelligence Scale for Children²⁵ or the Wechsler Adult Intelligence Scale.²⁶ All 15 participants with autism were cognitively able and yet severely impaired in their social functioning, as measured using the VABS-E¹⁹: a discrepancy of more than 3.5 SD was obtained between their verbal IQ and standard scores on the socialization domain of the VABS-E (see Table 1for a summary of participant characterization data). Participants with autism were individually matched for chronological age and verbal IQ with a comparison group of 15 adolescents and young adults recruited from the community and screened for history of major neurological or psychiatric illness. Given the typical variability in IQ profile in individuals with autism and the resultant complications when making decisions on matching procedures in studies of social functioning,^27,28 we adopted what is thought to be the most stringent approach to matching on the basis of verbal IQ rather than full-scale IQ. None of the participants had visual acuity deficits uncorrectable with eyeglasses.

All participants watched 5 digitized clips from the 1967 film version of Edward Albee's "Who's Afraid of Virginia Woolf?" This movie was chosen because it displays the intense interaction of 4 protagonists involved in a content-rich social situation likely to maximize viewers' monitoring of each person's socially expressive actions as well as those characters' reactions to the actions of others. The demanding social complexity in the movie mirrors complicated social situations that individuals with autism may encounter in everyday settings, such as a high-school cafeteria. This movie is also depleted of nonessential objects and events that might distract a viewer's attention from the social action. The clips ranged in length from 30 to 60 seconds depending on the content and duration of each chosen scene. The clips were separated from one another by 5 seconds of blank screen.

Eye tracking was accomplished using a dark pupil-corneal reflection video-oculography technique and hardware and software created by ISCAN Inc(Burlington, Mass). The system was head mounted and used a novel target-tracking method that enabled highly accurate eye tracking without having to restrain the participant's head (accuracy within ±0.3° over a ±20° horizontal and vertical range). The eye-tracking video equipment was mounted unobtrusively on the bill of a baseball cap. The participant's left eye was illuminated by a small, nonharmful infrared light–emitting diode. The movements of this eye were filmed using miniature imaging optics and a dichroic mirror. To obtain a frontal image of the eye without interfering with the participant's view, the eye-imaging camera was mounted in the bill above, facing downward and aimed at the angled dichroic mirror. The mirror, positioned in front of and below the eye, acted as a bandpass filter, reflecting infrared wavelengths (and thereby reflecting a frontal image of the participant's left eye) while transmitting wavelengths of the visible spectrum (allowing the participant a clear field of view). Directly underneath this mirror and in line with the eye-imaging optics was a miniature camera capped by a red additive filter. This camera filmed the scene in front of each participant. As the eye camera tracked movements of the pupil and corneal reflection, the scene camera recorded images of the participant's field of view. In a rectangular configuration around the computer screen were 5 red light–emitting diodes(1 at each of the screen's corners and 1 at the midpoint between the 2 upper corners) that provided reference points by which image-processing hardware(using data from the scene camera, aided by the red color filter mentioned previously) could identify the position and orientation of the "scene plane" even within a shifting field of view. In this way, the coordinates of the plane of the computer screen could be continuously tracked (and then calibrated with respect to the coordinates of the point of regard) despite any head or body movements. Eye-tracking data were fed from the cameras through ISCAN hardware at a rate of 60 samples per second and were recorded to videotape at the standard rate of 30 frames per second.

Participants sat in a comfortable armchair, 63.5 cm from a 48.3-cm computer screen mounted flush within a black wooden panel. The eye-tracking baseball cap was adjusted for participant comfort and clarity of view, and a brief calibration routine followed (consisting of having each participant look at a series of 5 points). Lights in the room were dimmed so that only images displayed on the screen could be easily seen, and the audio component was played through a set of concealed speakers. Data recording began after each participant reported an adequate level of comfort, an unobstructed view of the screen, and a clearly audible soundtrack.

Following each recording session, the videotape data were digitized and archived on computer. Five videotape clips lasting a total of 2 minutes 42 seconds were coded for each participant. Each frame of video was coded(30 frames per second, 4860 frames per participant). Figure 1 exemplifies a still image used for coding. For this illustration, the points of regard of 2 participants (1 with autism and 1 control) are superimposed onto a single frame. In practice, only one participant's data were coded at a time by 2 raters who were blind to the diagnosis and identity of each participant. Kappa coefficients²⁹ and percentages of agreement were calculated to assess interrater reliability, with κ of 0.82 and agreement of 87.2%, indicating "excellent agreement."³⁰ Each rater also recoded approximately 20% of the scenes chosen at random after at least 2 weeks. Test-retest reliability results were κ = 0.91 (agreement, 95.2%) and κ = 0.86 (agreement, 91.3%) for the 2 coders, indicating excellent levels of coding stability.

For each frame, the location of a participant's point of regard was coded as follows: a designation of "0" was assigned when the participant focused on the mouth region of the face, "1" was recorded when he focused on the eye region of the face, "2" was given when he focused on the body, and "3" was recorded when he focused on an object. In the event that a participant's point of regard was directly on the border between any 2 regions of interest, the 3 frames before and the 3 frames immediately after the frame in question were evaluated. The frame was then coded in accordance with the region into which the point of regard moved during the 7-frame sequence. To minimize such circumstances, region-of-interest resolution was maximized: only those shots in which the on-screen dimensions of the character's head were greater than or equal to 5° of the participant's field of view could be coded (this was the basis for selecting the 2 minutes 42 seconds of codable videotape data from the 5 clips shown to each participant; long-range shots were excluded). This criterion was used to ensure validity of the eye-tracking data coding, particularly in relation to coding of the eye vs the mouth region (if an on-screen head is smaller than 5° of the visual field, the system's margin of error is large enough to potentially cause notable discrepancies in the coding of facial feature preference). Finally, several scenes were discarded because there were no actors facing the camera or because the actors, the camera action, or both moved too rapidly for meaningful analysis of preferential viewing patterns.

Some frames that were coded could not be coded as fixation data. These included frames in which a participant blinked or made a saccadic shift. Blinks were defined by a loss of point-of-regard coordinate data (which was superimposed along with time code at the bottom of each picture) and then further verified by an inset videotaped image of the participant's eye (in such a case revealing closed lids). Initiation and completion of saccadic movements were defined by a rotational velocity threshold of 30° per second,³¹ which translated in our experimental setting into a linear velocity of 15 or more pixels per frame (ie, d≥15 pixels, where d = √(x²) + (y²), over the course of 1 frame [33 milliseconds]). Saccadic movements were also confirmed by the inset eye image. In addition, some frames could not be coded for the following reasons: off-screen fixations(watches, hands, etc), rubbing of eyes, and obstruction of the scene view camera (hand in front of face, nose rubbing, etc). These frames, recorded as "no data," amounted to a mean (SD) 5.2% (6.6%) of total time for controls and 15.6% (15.9%) of time for viewers with autism (P= .03). Although this significant difference could be related to decreased concentration and increased distractibility in participants with autism, we cannot infer any specific conclusions from this statistical difference because other, non–attention-related factors could also be at play, including, for example, rubbing the eyes or sneezing. Future studies will need to pay more attention to coding schema for characterization of lost data.

T tests with Bonferroni corrections for number of between-group comparisons were used to test group differences in the percentage of total viewing time spent on fixations on mouth, eye, body, and object regions of the scene images. Pearson correlations were used to explore the degree of the relationship between fixation patterns and the 2 measures of social competence—level of social adjustment (operationalized as the age-equivalent score on the socialization domain of the VABS-E) and degree of autistic social impairment (operationalized as the total algorithm score on the social section of the ADOS).

Consistent with our predictions, individuals with autism focused 2 times more on the mouth region, 2 times less on the eye region, 2 times more on the body region, and 2 times more on the object region relative to age- and verbal IQ–matched controls (Table 2). Effect size was greatest for fixation on the eye region, making it the best predictor of group membership (d = 3.19). There were no significant correlations between any of the measures of fixation time and chronological age or verbal IQ in the 2 groups except for a positive significant correlation between percentage of fixation time on the mouth region and age in the group with autism (r = 0.62; P = .01). When one outlier (a participant with autism aged 38 years) was removed from the sample, this correlation was no longer significant. None of the other correlations were altered by excluding this person. As shown in Figure 2, there was little overlap in the measures of fixation time across the 2 groups, although there was some variability within the groups, except for fixation on objects in the control group.

We next explored the association between fixation time measures and measures of social competence. Contrary to our expectation, fixation time on the eye region was not associated with either social adaptation (VABS-E socialization scores) or social disability (ADOS social scores) (r = −0.20 and r = 0.14, respectively). In contrast, fixation times on the mouth region (Figure 3) and on the object region (Figure 4) were strong predictors of social competence, albeit in different directions. Fixation time on the mouth region was associated with greater social adaptation (ie, more socially able) and lower autistic social impairment (ie, less socially disabled). Going in the opposite direction, fixation time on the object region was associated with lower social adaptation and greater autistic social impairment. Fixation time on the body region followed the same trend as for the object region, but the correlations were not significant(r = −0.49 and r =0.34 for social adaptation and social disability, respectively). When the outlier (a participant with autism aged 38 years) was excluded from correlational analyses, none of the measures of association were altered except the positive correlation between fixation time on objects and the measure of social adaptation, which was previously significant at P<.10 and was now significant at P<.05.

We found significant differences in percentages of visual fixation time on mouth, eye, body, and object regions when viewing naturalistic social situations among cognitively able adolescents and young adults with autism relative to age- and verbal IQ–matched controls. The best predictor of group membership was percentage of fixation time on the eye region: the control group visually fixated on the eye region 2 times more than did the group with autism, and there was no overlap in the distribution of results. However, within the group with autism, this variable was unrelated to outcome measures of social competence. In contrast, percentages of fixation time on the mouth and object regions were strong predictors of social competence measures, with higher mouth fixation time associated with higher levels of social adaptation and lower levels of autistic social impairment, and object fixation time associations going in the opposite direction.

This study shows the utility of developing performance-based paradigms capable of quantifying social functioning under more naturalistic conditions. The effect sizes obtained for between-group comparisons were markedly larger relative to experimental studies³² of social functioning in autism and more comparable to the few studies¹⁴ available that focused on spontaneous responses rather than explicit problem-solving tasks. Similarly, the enhanced ecological validity aimed for in the present study also led to stronger associations between performance-based experimental measures and outcome measures of social competence, an important goal to be achieved in the search for dimensional endophenotypes for genetic research.¹⁷ More generally, the paradigm presented herein provides a valuable window into the ways individuals with autism search for meaning when confronted with complex social situations.

Given that this is the first study of its kind, comparisons with other studies can only be indirect. Nevertheless, the small number of available studies is generally supportive of our findings. For example, some studies have suggested increased reliance on mouths rather than eyes when individuals with autism are required to perform face perception tasks,^5,6 great difficulty in "reading" the meaning of eye expressions,⁷ and increased orientation to objects relative to people.^4,8 A recent study by Joseph³³ addressed the differential reliance on eyes and mouths in autism more directly than previous studies. When 10- to 12-year-old, cognitively able individuals with autism were asked to perform a match-to-sample face recognition task in which faces varied as a function of eye or mouth changes, they exhibited a selective advantage for mouths.

Although the present study was not designed to explore the factors underlying visual fixation patterns in autism, the strong predictive association between mouth and object, but not eye fixations and outcome measures of social competence, suggests that specific hypotheses need to be pursued in future, more-refined studies. The finding that a higher percentage of mouth fixation time predicted a higher level of social competence is as intriguing as it is counterintuitive, and, therefore, it deserves special attention. Given the well-known association between level of verbal skills and better outcome in autism,³⁴ it is possible that the participants in our study were focusing on mouths because that is where speech comes from. By concentrating their efforts on something that they can understand, they might attain better understanding of social situations. Nevertheless, such a compensatory strategy is not without its limitations given that the meaning of language is often modified by nonverbal social cues such as eye expressions. The fact that percentage of time focused on eyes was unrelated to measures of social competence suggests the possibility that for individuals with autism, looking at eyes does not accrue considerable advantages in their efforts to understand social situations, or, in other words, that the eyes are not meaningful to them.^7,34

The hypothesis that increased fixation on mouths results from concentration on speech needs to be further refined on the basis of what we know about visual attention to facial regions in speech perception tasks.³⁵ Whereas audiovisual speech perception (like the stimuli in the present study) draws gaze toward the talker's eyes,³⁶ speech reading (ie, seeing someone speak with no accompanying audio) is more likely to draw gaze toward the talker's mouth.³⁷ However, visual fixations in speech reading depend on the nature of the task. If participants are asked to perform a segmental speech perception task (ie, to guess what the talker is saying), the viewer is more likely to focus on the mouth region.^36-38 However, if participants are asked to perform a prosodic speech perception task (ie, to guess the inflection of the talker's voice, eg, sad, happy, or mad), then the viewer is more likely to focus on the upper region of the face (eyes and eyebrows).³⁷ On the basis of this literature, our visual fixation results could suggest that our participants were not only focusing on the verbal content of speech but were also ignoring paralinguistic cues such as prosody,³⁹ which are usually essential to understanding nonliteral aspects of social situations such as intentions and attitudes.⁴⁰ In other words, our findings could be related to the overreliance of individuals with autism on the literal aspects of speech at the expense of intonational cues associated with social meaning.⁴¹ By pairing speech perception tasks with eye-tracking measures,³⁷ these relationships are likely to be clarified.

Another line of inquiry for future studies relates to the possibility that increased fixation on mouths and decreased fixation on eyes indicate that individuals with autism may acquire a degree of perceptual expertise on mouths but not on eyes. Findings of the previously described study by Joseph³³ seem to point in this direction. When the face recognition task was presented in the form of inverted faces, children with autism did not show the typical decrement in performance⁴²(the so-called inversion effect⁴³) in the eye condition, but did so in the mouth condition. Given that an increase in the magnitude of the inversion effect and associated transition from feature-based to configural processing marks the development of perceptual expertise relative to a class of objects,^44-47 the findings of Joseph raise the possibility that children with autism have expertise when recognizing faces that vary in mouth features but not when they vary in eye features. Future studies that could manipulate these variables more explicitly in still (ie, photographs) and dynamic (ie, videotape) stimuli might clarify these issues. Also, given the recent surge of neuroimaging research on the brain circuitry involved in the acquisition of perceptual expertise,⁴⁷ the coregistration of neurofunctional and eye-tracking data might prove to be particularly useful in elucidating the unusual patterns of behavioral and brain functioning in regard to face perception in autism.^16,18,48

A final line of inquiry suggested by our results concerns the need to explore the association between more fixation time on objects and decreased social competence in participants with autism. This finding is consistent with the notion that focus on objects means reduced salience of social stimuli^4,15 and, therefore, decreased likelihood of understanding the social situation. The percentage of fixation time on objects was small relative to fixation time on the face (mouth and eyes). This is not surprising given that our videotape clips were deliberately chosen to minimize inanimate distractions. Future studies may maximize the predictive utility of the association between fixation on objects and outcome measures of social competence by manipulating the prominence of inanimate stimuli (eg, by making objects move or by using objects known to attract the attention of individuals with autism). Clinical observations have shown the devastating impact of object-rich environments on the ability of children with autism to focus on socially meaningful aspects of educational environments.⁴⁹

The present study has several limitations. First, although close to three quarters of all individuals with autism have a degree of mental retardation,⁵⁰ we studied more cognitively able individuals with autism. We do not know if these results will extend to participants with lower IQs. Neither do we know what trends to expect in younger children with autism or in individuals with milder manifestations of the condition (eg, Asperger syndrome and Pervasive Developmental Disorder–Not Otherwise Specified[also called "atypical autism"]). Second, our design does not rule out possible contributions from abnormal functioning in areas other than social visual pursuit (eg, underlying attentional or perceptual abnormalities).^51,52 The use of more traditional eye-tracking protocols focused on the integrity of brain mechanisms associated with eye movement⁵³ alongside our novel application of this technology will be necessary to rule out explanations other than the ones explored in this study. Third, although the measures reported in this study were informative, they are unlikely to be the most sensitive indicators of abnormalities in social visual pursuit in autism. In a case study reported elsewhere,⁵⁴ we illustrate a series of social visual tracking phenomena that seem to capture the abnormalities in autism in a much more dynamic and stark fashion than summaries of visual fixation times. The moment-by-moment visual traces left behind by the saccadic movements and fixations of individuals with autism illustrate more vividly their atypical attempts to create social meaning out of what they see. Perhaps paradigms capable of quantifying the subtleties of such data will soon emerge from these as-yet only heuristic approaches.

Submitted for publication June 14, 2001; final revision received October 10, 2001; accepted October 24, 2001.

This work was partly supported by grants P01 HD 03008 and P01 HD/DC35482 from the National Institute of Child Health and Human Development, Bethesda, Md, and by a grant from the National Alliance for Autism Research, Princeton, NJ.

This study was presented in part at the annual meeting of the National Institute of Child Health and Human Development, Collaborative Projects of Excellence in Autism, New Haven, Conn, May 2, 2001.

We thank the participants and their families; Elizabeth Levitan, BA, Tammy Babitz, MA, Sanno Zack, BA, Catalina Hooper, BA, Amy Augustyn, BA, and Christopher Abildgaard, BA, for their assistance with this study; and Katarzyna Chawarska, PhD, for her contribution to interpretation of findings. We also thank Warner Bros, Elizabeth Taylor, George Segal, and the estates of Sandy Dennis and Richard Burton for permission to use their images.

This article is dedicated to the memory of our mentor, collaborator, and colleague, Donald Cohen, MD, whose legacy embodies the best in clinical services, public advocacy, clinical science, bioethics, and mentoring.

Corresponding author and reprints: Ami Klin, PhD, Yale Child Study Center, 230 S Frontage Rd, New Haven, CT 06520 (e-mail: ami.klin@yale.edu).