Durations shown as the time from the initiation of movement to maximal excursion (light green), the time that maximal excursion was maintained (dark green), and the return from maximal excursion to a neutral position (green). The lowest bars represent the delay increments introduced to one side of the face in the perceptually assessed video clips. Inset provides an example of blink delayed on the left by 3 increments (100 milliseconds) relative to the right.
Examples are from the symmetrical (baseline or nondelayed) video version, including a representative neutral position (A), eyebrow raise (B), blink (C), smile (D), and lip depression (E). Movements B-E represent maximal movement excursions, represented by the black bars in Figure 1. Slow and fast eyebrow raisings had similar maximal excursions and are represented by a single still frame (B).
Movement reported as a percentage of participants (N = 58) for 5 movements across 7 levels of potential side-to-side difference in movement timing, ranging from 0- to 264-millisecond delay. For all movements, video clips with 0-millisecond side-to-side delay were typically correctly reported as being symmetrical, and 99 milliseconds of timing difference was typically identified as not symmetrical. The perception of movement symmetry differed by movement at delays of 33 milliseconds and 66 milliseconds. The dashed line at 50% represents the detection threshold.
For each of the 5 perceptually assessed movements, naturalness scores consistently ranked as less natural (higher value) with increases in side-to-side asymmetry of movement.
Note how blink is perceived as relatively unnatural within the first several levels of delay compared with the other movements and that slower movements, such as smile and slow eyebrow raising, are generally perceived as more natural across multiple levels of asymmetry compared with faster movements.
Kim SW, Heller ES, Hohman MH, Hadlock TA, Heaton JT. Detection and Perceptual Impact of Side-to-Side Facial Movement Asymmetry. JAMA Facial Plast Surg. 2013;15(6):411-416. doi:10.1001/jamafacial.2013.1227
In this study, we examined whether specific facial movements have different time-delay detection thresholds, and to what extent such side-to-side facial movement asymmetry affects subjective ratings of movement naturalness. Ratings of dynamic asymmetry in experimentally manipulated video recordings demonstrate that there are different side-to-side time-delay thresholds for distinct regions of the face, with a strong inverse correlation between naturalness rating and the length-of-time delay. These findings will be helpful for counseling patients with unilateral facial paralysis and guide the design of neural interfaces for facial reanimation.
To determine the detection threshold of side-to-side facial movement timing asymmetry and measure its effect on perceived movement naturalness.
Design, Setting, and Participants
Videos of 5 symmetrical facial movements (eye blink, rapid eyebrow raising, slow eyebrow raising, smiling, and lip depression) were edited to introduce 6 levels of side-to-side timing asymmetry, ranging from 33 to 267 milliseconds. Participants (N = 58) viewed video clips through an online survey service, indicating whether they noticed side-to-side asymmetry and judging movement naturalness on a 5-point scale.
There was a significant difference among facial movements in asymmetry detection threshold. There was a strong correlation between naturalness ratings and amount of delay across movements (R = 0.823), with greater asymmetry being judged as progressively less natural. Blink was judged as less natural at 33, 67, 100, and 133 milliseconds of side-to-side delay compared with all other movements (P < .005).
Conclusions and Relevance
Side-to-side asymmetry in blink timing is detected sooner and viewed as less natural compared with asymmetry of the eyebrow and lips. At 100 milliseconds of delay, nearly all movements are detected as asymmetric, although blink is judged as the least natural. These findings will help set timing goals for facial pacing technologies treating unilateral paralysis.
Level of Evidence
Facial paralysis is a devastating condition; afflicted patients experience functional deficits, aesthetic issues, and profound psychosocial challenges.1- 3 In addition to traditional surgical interventions involving nerve and muscle transfers, some authors have proposed neural prosthetic approaches that could reanimate the face by electrically stimulating facial nerves and/or paralyzed muscles.4,5 The most promising application of functional electrical stimulation (FES) for facial reanimation would involve instances of unilateral paralysis, in which one side of the face has normal movements that can be detected and used to “pace” stimulation of the contralateral paralyzed face. Although theoretically feasible, this approach harbors several technological challenges in achieving rapid, accurate detection and subsequent FES reanimation of movements.
To make informed decisions regarding implementation of neural prosthesis components (ie, weighing the trade-off between system component speed vs invasiveness), the threshold for detection of side-to-side movement asymmetry and its perceptual effect on movement naturalness must be established. Potential hardware interfaces with the body, for first detecting and then eliciting facial movements, introduce varying degrees of delay in this sequence. For example, electromyographic-based detection of blink can occur simultaneously with (or even precede) visible movement,6 but bioelectric recordings are prone to signal interference from adjacent muscles, which could be difficult to distinguish and eliminate. Other potentially more accurate detection strategies, such as accelerometry or image-based monitoring, may introduce delay, causing conspicuous side-to-side asymmetry in facial movements. On the stimulation side, implanted electrodes generate relatively fast movement, while less invasive body surface stimulation may introduce delay that might affect perceived gesture naturalness. Although studies have examined the degree of lateral asymmetry detectable in static images of the face,7,8 the detection thresholds and perceptual impact of dynamic timing asymmetry has not yet been determined.
The goal of this study was to quantify the amount of time delay between movements on the left and right sides of the face that causes observers to detect asymmetry in facial movement. We also examined whether certain specific facial movements (ie, eye blink, eyebrow raise, smile, or lip depression) had different time delay detection thresholds and investigated to what extent side-to-side facial movement asymmetry affected subjective rating of movement naturalness. Taken together, these data will help guide the design of neural interfaces for facial reanimation.
We performed a prospective perceptual assessment of facial movement video clips presented to participant volunteers through an Internet website (Surveygizmo.com; Widgix, LLC). The video clips were recordings of a 26-year-old woman who provided informed consent under a study protocol approved by the Massachusetts Eye and Ear Infirmary Institutional Review Board.
Video recordings of facial movements were obtained with the woman seated upright in a clinical examination chair, with the back of her head resting firmly against a padded headrest to minimize head movements. Recordings were made under even illumination (StudioMax II 320 umbrella lights; Photogenic Inc) using a digital video recorder (Canon HF200, 1080 pixels, 2892 kilobytes per second, 29.97 frames/s; Canon) while the woman repeated 5 particular facial movements including eye blink, rapid and slow eyebrow elevation, smiling, and lip depression. Recordings were reviewed to select a single representative rendition of each movement type, and the digital video files were edited (VideoStudio Pro X3; Corel) to make them uniformly 3 seconds in length. Each video clip had at least 467 milliseconds of neutral posture preceding each movement. Figure 1 presents the durational aspects of each movement, including total duration, latency from start to maximum excursion, duration of peak excursion, and return to a neutral facial posture.
Facial movement video clips were edited to ensure complete side-to-side symmetry of movement by splitting the screen down the vertical midline of the face and duplicating one side of the face as a mirror image onto the opposite side. These symmetrical (baseline) videos were edited to create additional video clip versions with one side of the face delayed 1, 2, 3, 4, 6, and 8 frames relative to the other side (29.97 frames/s, generating approximately 33.34 ms/frame of introduced side-to-side delay). Moreover, the side of the face that was delayed was randomly chosen for each degree of delay. A white strip 10 pixels wide was placed on the face midline on all videos to obscure the right-left break in image continuity that may otherwise appear when one side was delayed relative to the other side (Figure 2).
Perceptual assessment participants were sent an e-mail with a brief description of our study goals and a link to the survey website. They were informed that the survey would require approximately 15 minutes to complete (based on pilot trials) and that participation was voluntary and anonymous. No remuneration was provided for survey completion. The link opened an introduction page in the participant’s default web browser, providing a study description and directions. The second page of the survey presented a demonstration video showing symmetrical (baseline) versions of all movements that would be presented in the test. Also presented on this page was an example of the 5-point Likert scale of “natural” to “unnatural” that participants were to complete for each video if they indicated seeing side-to-side asymmetry in movement timing and a pull-down menu for participants to indicate what device they were using to complete the survey (laptop, desktop, tablet, or smartphone), their sex, and their age.
After the introductory pages, participants viewed the 35 video clips one at a time in random order, clicking a “next” button after reporting whether they detected side-to-side delay, and then rated movement naturalness. Video files were streamed on the survey platform from an online media service (YouTube LLC) with a frame size of 700 × 576 pixels in the web browser. The media service provided video data buffering that allowed playback on devices with a wide range of processing capabilities and Internet bandwidth without interruption or distortion. If participants encountered playback discontinuity, they were able to repeat the viewing with the use of the video window control panel.
Test results were downloaded from the survey service to a spreadsheet (Excel, Microsoft Corp). The results then were screened for outlying respondents.
The relationship between movement asymmetry and naturalness ratings was statistically examined with the Friedman nonparametric test, which generated a rank ordering for each participant based on Likert scale values (perception of naturalness) vs the 7 levels of split-screen delay within each movement, and compared rankings among participants in relation to a χ2distribution. A Wilcoxon signed rank test was used to compare naturalness scores for each movement vs all other movements (10 comparisons) within each level of split-screen delay. A stringent α level of P < .005 was used for each set of 10 pairwise comparisons to mitigate α inflation. Post hoc analysis with the Wilcoxon signed rank test was performed to examine differences between degrees of frame delay within movements, using a Bonferroni correction to maintain an aggregate α level of P ≤ .05 for rejecting the null hypothesis.
Sixty-three recipients of the e-mail request for participation in this study completed the video assessment survey. Five of these respondents (8%) had correlations between the number of frames delayed and naturalness ratings that were more than 2 standard deviations lower than the average correlations from all respondents combined. It was assumed that participants would perceive movements with greater side-to-side timing asymmetry as less natural; therefore, these 5 respondents were considered outliers and were removed from the survey results, leaving 58 contributing participants. There were 33 female and 25 male participants, with ages ranging from 19 to 73 years (mean, 41.2 years). There were no significant differences between men and women, as well as no significant differences across age groups (when divided into 5 groups, approximately by decade). Participants reported taking the perceptual test on a desktop computer (30 [47%]), laptop (28 [45%]), tablet (3 [5%]), or smartphone (2 [3%]), with no clear difference in test outcome based on device (although small group sizes for tablet and smartphone precluded statistical examination).
Participants were presented with the facial movement video clips one at a time in random order. When there were no frames of delay between the 2 sides of the face, a mean of 95% of the participants indicated that the face moved symmetrically across the 5 movements (Figure 3). There was a significant difference across facial movements when one side was delayed by 33 milliseconds (1 frame) relative to the other side (χ24 [n = 58] = 15.87; P < .01), wherein 81% of participants perceived facial movement as symmetrical for all movements combined except for blink, which was perceived as symmetrical by only 33% of the participants. There was also a significant difference across facial movements when there was 67 milliseconds of split-screen delay (2 frames; χ24 [n = 58] = 26.31; P < .01). With 67 milliseconds of asymmetry, blink was rarely (2%) perceived as symmetrical, whereas the other movements were perceived as symmetrical in a range of 12% to 43% (Figure 3). With 100 milliseconds (3 frames) of relative delay, a mean of only 3% of participants judged the movements as symmetrical.
When a video was judged as symmetrical, the naturalness score was assigned a value of zero. When a participant reported seeing asymmetry, the survey webpage presented a 5-point Likert scale for judging facial movement naturalness, ranging from 1 (natural) to 5 (unnatural). There was a strong mean correlation between naturalness ratings and amount of delay across movements (mean [SD] R = 0.823 [0.04]), with greater asymmetry in split-screen movement being judged as progressively less natural. There was a statistically significant difference in the ranks of the delay amount for all movements (5 tests: χ26 [n = 58] = 258-281; all P < .001), confirming the effect of delay on perceived naturalness. Rankings were averaged across participants and are presented in Figure 4, revealing a consistent relationship between split-screen time delay and naturalness score for each movement.
Naturalness scores differed according to movement type within multiple levels of split-screen delay (Figure 5). A Wilcoxon signed rank test revealed that blink significantly differed from all other movements at 33-, 67-, 100-, and 133-millisecond delay (1-4 frames, respectively; P < .005). At 200-millisecond delay, blink differed from slow eyebrow, and at 267-millisecond delay, blink differed from slow eyebrow and fast eyebrow. Finally, at 67 milliseconds of delay, slow eyebrow differed from fast eyebrow and lip depression.
Pairwise comparisons were performed to determine whether each level of split-screen delay significantly differed in median naturalness score among participants from all other levels within each movement. These post hoc analyses were conducted using the Wilcoxon signed rank test with a Bonferroni correction, resulting in a significance level of P < .0024 (significance level of .05/21 comparisons within each movement). For each compared level of split-screen timing asymmetry, participants typically judged the more symmetrical video as being more natural. The only pairs for which this pattern was not significant are listed in the Table.
Significant facial asymmetry has been linked to the perception of disfigurement,9,10 often causing patients to report higher levels of depression11 and lower quality-of-life scores.12 These facial asymmetries can result from facial paralysis, lesions, or other changes to the face. Photographs of individuals with facial paralysis have been ranked as less attractive than photographs of individuals without facial paralysis,8 and typical visual scan-path patterns are redirected toward the side with facial paralysis,7 a crooked nose,13 or a facial lesion.14 Surgical reconstruction and reanimation techniques can potentially address the causes of facial asymmetry, and it is important to know the relationship between side-to-side difference in movement range and/or timing and the resulting perception of movement naturalness. This is particularly true for the development of facial movement pacing systems, since the implementation of different hardware solutions for movement detection and elicitation can introduce different amounts of side-to-side timing asymmetry, and the perceptual impact of timing delays must be considered in the early stages of system design.
One of the most commonly used techniques for quantification of surgical outcomes is measurement of recovery of excursion in the involved facial region.15- 17 Paradoxically, Paletz et al18 have shown that in individuals whose faces were considered normal, the amount of variation of oral commissure excursion between each side can be as much as 52%. Although the investigators observed varying distances of excursion, the vector of excursion was consistently symmetric in smiles of this population. This suggests that the vector of excursion is just as important as the net distance of excursion when one aims to achieve natural facial movement through FES or/and surgical techniques in patients with unilateral facial paralysis. Oftentimes, only net excursion has been used in published studies,19- 21 likely because spatial measurement is reliable and objectively quantifiable. However, for more comprehensive analysis of facial evaluation, one should consider the dynamic components of facial movement, such as synchronicity and symmetry of movements.
Our study demonstrates that there are different time-delay thresholds for detection of dynamic asymmetry in each distinct region of the face, with subjective naturalness rating and the length of time delay demonstrating a strong correlation. Specifically, pacing of blinks apparently needs to occur with less than 33 milliseconds of side-to-side delay to remain below the detection threshold. Moreover, blinks were perceived in our study as significantly less natural than all the other gestures across the first few levels of delay, likely stemming from their fast movement and relatively short duration. Slower movements, such as slow eyebrow raise and smile, were more often perceived as symmetrical at 33 and 67 milliseconds of side-to-side asymmetry and were generally perceived as more natural at these time delays compared with the other movements. These findings may be expected, since unilateral delays of faster movements will generate greater side-to-side differences in facial appearance in less time than delays in slower movements, and when the movements are brief, the direction of rapid movements become out of phase with even small amounts of side-to-side delay (ie, blink) (Figure 1).
To our knowledge, the present study is the first to examine the detection thresholds of side-to-side timing asymmetry in dynamic facial movements. In addition to the differences in movement speed discussed above, there are further possible explanations for why certain movements had different detection thresholds. For example, eye-tracking studies demonstrate that when individuals look at human faces, they follow a scan path forming a triangle among the eyes, nose, and mouth.22 Moreover, a recent eye-tracking study found that when healthy faces were portraying neutral, angry, or sad expressions, there was greater focus on the eyes than on other regions of the face.23 Therefore, the neutral baseline start of each video may have predisposed observers to focus on the eyes and detect differences in their movement relatively more quickly than movements in other facial regions.
There are known left-to-right facial asymmetries in dynamic expressions based on what emotion is being portrayed,24 and it is common to encounter side-to-side expression timing differences across and within individuals, as described by Borod et al.25 As a result, it is likely that people ordinarily expect greater degrees of timing asymmetry in some expressions compared with others. For example, healthy individuals do not typically exhibit asymmetrical blink but may exhibit an asymmetrical smile or eyebrow raise. Therefore, it is possible that observers usually have a higher detection threshold and more tolerance for asymmetry in these latter expressions, consistent with the present findings.
Perceptual evaluations of movement asymmetry as presented in this report are a useful starting point for setting facial pacing timing goals but may require further study given the complexity and nuances of facial movement perception. Although we observed consistent differences in the detection and perceptual impact of timing asymmetries for facial movements spanning multiple zones, we did not determine which aspects of side-to-side differences bring about the perception of asymmetry and to what degree different movement rates within each facial zone affect naturalness perception. Moreover, future work is needed to determine whether the amount of excursion during facial movement, in the absence of timing considerations (eg, synchronicity or velocity), demonstrates the same strong inverse correlation between degree of asymmetry and perception of naturalness. Some of these answers might best be attained through performance testing of facial pacing systems as they are developed, but the present findings provide an indication of movement latencies that those initial systems should strive to achieve.
Accepted for Publication: February 20, 2013.
Corresponding Author: Tessa A. Hadlock, MD, Department of Otolaryngology, Massachusetts Eye and Ear Infirmary, 243 Charles St, Boston, MA 02114 (firstname.lastname@example.org).
Published Online: August 8, 2013. doi:10.1001/jamafacial.2013.1227.
Author Contributions:Study concept and design: All authors.
Acquisition of data: Kim, Heller, Hadlock, Heaton.
Analysis and interpretation of data: Kim, Heller, Hadlock, Heaton.
Drafting of the manuscript: Kim, Heller, Hadlock, Heaton.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Heller, Hadlock, Heaton.
Obtained funding: Hadlock, Heaton.
Administrative, technical, and material support: Kim, Hadlock, Heaton.
Study supervision: Hohman, Hadlock, Heaton.
Conflict of Interest Disclosures: None reported.
Previous Presentation: This study was presented as a poster at the Combined Otolaryngological Spring Meetings; April 12, 2013; Orlando, Florida.