Measurement of Facial Movement With Computer Software

Objective  To adapt desktop computer software to objectively grade facial movement.

Design  The criteria of the facial nerve grading system by House and Brackmann, the current "gold standard," are prone to ambiguous interpretation. Proposed objective grading systems compare the movement of points on each side of the face or use subtraction and thresholding of digitized images of the face to yield images that represent moving areas of the face. The movement of a point on the face and the area of motion determined by digital subtraction were compared during an increasing smile in healthy subjects. The Nottingham system (calculated using measurement of the movement of 4 points on the face) using desktop computer software (Adobe Photoshop 3.0, Adobe Systems Inc, Mountain View, Calif) to measure movement of the points was compared with the system by House and Brackmann. The computer software was used to subtract digitized images and derive a facial movement score, which was compared with the scores of the systems by Nottingham and House and Brackmann.

Setting  Academic otologic practice.

Study Participants  Nine patients with varying degrees of facial nerve disability and 7 individuals with normal facial nerve function.

Results  The movement of the oral commissure compared with the apparent area of movement of the face determined by digital subtraction had high intersubject variability. In patients with facial weakness, the Nottingham score had a correlation coefficient of −0.97 compared with the House and Brackmann grade, and the digital subtraction score had a correlation coefficient of −0.62 (paired Student t test).

Conclusions  The desktop computer software can be used to calculate the Nottingham score. Digital subtraction as a measure of facial function warrants further study.

IN 1985, Pearson¹ wrote of facial nerve grading: "Subjectivity cannot be eliminated, and so complex a function cannot be readily simplified." To a great degree his words have proved true, although many have worked to make the process of facial nerve grading objective.

Ideally, a facial nerve grading system should be continuous (rather than discontinuous, where the results are divided into discrete categories), easy to use, and unambiguous. Using such a system, reports of outcomes of surgery (eg, facial nerve injury following acoustic neuroma surgery) or interventions (eg, steroid use in Bell palsy) would be more easily evaluated and observer bias would be reduced. House and Brackmann's² (H-B) system, the most common and easy to use system, is discontinuous, subjective, and ambiguous.

Other systems have been proposed to replace the H-B system. Burres and Fisch^3,4 developed a system based on a study of facial biomechanics. While it is objective and continuous, this system is time-consuming and cumbersome.^5,6 Critics⁶ of the system have also noted that it tends to underestimate the degree of dysfunction in severe paralysis and overestimate it in mild paralysis.

In 1994, Murty et al⁶ described the Nottingham system of measuring facial nerve function. It is calculated by measuring the movement of 4 points on the face and comparing the abnormal side with the normal side as a percentage. It gives a global percentage scale that is continuous and has been validated by the Murty et al against the H-B system. In the Nottingham system, issues of secondary defects are addressed in parts 2 and 3 of a 3-part score in a manner similar to the TNM scale used in oncology. Figure 1 shows the reference points measured in the Nottingham system, and Figure 2 shows how the Nottingham score is derived.

In 1992, Neely and associates⁷ described a computerized image analysis method. By subtracting digitized video images of the face in motion from an image of the face at rest, they derived an image that makes areas of the face that change their gray scale value appear bright. By thresholding (converting all pixels above a certain value to white and those below the value to black), areas that appear to approximate the parts of the face in motion are highlighted in white and still areas are black. With analysis of sequential video images, Neely et al derived intensity-duration curves. They found that a score derived using this analysis correlated with the H-B score. To date, the software used in their study has been unavailable to the public.

In 1996, Neely and associates⁸ updated their grading system by adding a subjective observation of specific areas of the face (brow, eye, and mouth).These observations are limited to 4 categories (normal, less than normal without facial dyskinesia, less than normal with facial dyskinesia, and no movement). The subjective component is then added to a 2-part computer-derived score for each area of the face to arrive at a composite index. Using the pixel subtraction method, the first part of the computer-derived score assigns the area to 1 of 5 types of intensity-duration plots. The second computer-derived score is based on the maximum amplitude of the computed curve. The scores are added by region and summed to arrive at a composite index for each half of the face. Interestingly, the subjective component is given the most weight (>80%) in the composite index, negating the benefits of a computerized objective grading scale and making this a primarily subjective grading scale.

We sought to develop a computer-based method of objective analysis. Our goals were to use widely available, reasonably inexpensive software for a desktop system to objectively grade facial movement. We also sought a method that would be facilitated by but not require the use of a computer. We also wished to explore the possibility of pixel subtraction to analyze facial movement.

The computer software (Adobe Photoshop 3.0, Adobe Systems Inc, Mountain View, Calif) is the industry standard for computerized image manipulation, chiefly used by graphic artists. Among its many features, Photoshop allows image subtraction and the counting of pixels in areas of images. After determining that pixel subtraction could be performed using Photoshop, we studied the relationship between a 2-dimensional movement of a point on the face and the number of pixels derived by the pixel subtraction method in that area of the face in control subjects. We then studied a group of patients with facial nerve problems. We compared point-to-point movement analysis to grade facial nerve function with the pixel subtraction method.

Seven subjects with no history of facial nerve problems were studied as a control group. Nine subjects with varying degrees of facial nerve function from a variety of causes were studied. Written permission for the use of facial photographs in publications was obtained from all individuals.

A still camera (Canon EOS 650, Canon, Lake Success, NY)fitted with a ring flash was used to photograph the subjects. The camera was fixed to a slitlamp examination table that stabilized the head to keep a constant relationship between the camera and subject. Images were transferred to CD-ROM (Kodak PhotoCD, Eastman Kodak Co, Rochester, NY). Photoshop running on a computer (Macintosh Centris 650, Apple, Cupertino, Calif) was used to perform image analysis.

The 7 control subjects used to study the relationship between the distance moved by a point on the face and the area represented by the pixel subtraction method were photographed in a darkened room. Faces were illuminated only by the ring flash. Adhesive marks were positioned at the oral commissure, the lateral canthus, the inferior orbital rim, and the superior orbital rim (Figure 1). A sequence of photographs was taken of a progressively increasing smile, starting at rest.

After transfer to CD-ROM, the image of the face at rest (Figure 3, A) was viewed using Photoshop at a resolution of 512×768. The x and y coordinates of the marker at the oral commissure were then read using the Info Palette. An image of the smiling face (Figure 3, B) was then placed on top of the face at rest. Using the Layers palette, the overlaid image was made semiopaque (Figure 3, C) and placed so that constant reference points (in this case, the corneal light reflex as well as the rulers on the slitlamp examination table) were aligned (Figure 3, D). The overlaid image was then made opaque and the x and y coordinates of the marker at the oral commissure were read using the Info Palette. Using a spreadsheet (Microsoft Excel, Microsoft Corporation, Redmond, Wash), the movement of the marker was calculated using the Pythagorean theorem. Figure 3, E, shows how position of the markers can be calculated precisely using Photoshop.

Pixel subtraction was performed using Photoshop by first converting the color images to gray scale with the Image-Adjust-Desaturate command. The image of the smiling face was then laid on top of the face at rest and aligned as described previously. After alignment, the images were subtracted using the Difference command in the Layers palette. After clearing the Marquee tool, the result was subjected to thresholding using the Image-Map-Threshold command. A threshold of 30 was selected. Like Neely et al,⁷ we found little difference in the result when different threshold levels were used. Figure 4 shows the result of these image manipulations for the images used in Figure 3. The number of pixels in a given area can be determined by outlining the area of interest with the Marquee or Lasso tool and then counting the white pixels using the Image-Histogram command. This last step was performed 3 times, and the average was used in analysis.

Subjects with impaired facial nerve function were photographed in a manner identical to that used for the control subjects. Three standard facial movements were studied: full smile, eye closure, and forehead wrinkling. The position of the facial markers was calculated by the method described and the Nottingham score calculated using the spreadsheet. On the identical images, the pixel subtraction method was used. The number of pixels in 3 zones (Figure 5) was determined for each half of the face. The sum of the pixels in zones 1, 2, and 3 of the impaired side was divided by that on the normal side and the result expressed as a percentage of relative movement.

Figure 6 shows how 2-dimensional movement of a marker of the face corresponds to the area of movement derived by the pixel subtraction method.

We found that while progressive movement of the marker at the corner of the mouth was associated with an increase in the pixel area, the variation between individuals made generalization of a relationship between the 2 variables impossible. We found that there was little overall side-to-side variability. We observed that subjects with facial hair or coarse skin were among those with the steepest slopes, suggesting that highly textured surfaces may increase the apparent area of movement.

To determine if movement in the anterior-posterior direction (z axis) not measured by our 2-dimensional photographic technique may explain the variability between the movement of the marker and the result of pixel subtraction, the movement of the oral commissure on the x, y, and z axes was manually measured for 6 control subjects. The relationship between the movement on the x and y axes and the z axis was linear with a regression coefficient of 0.92 (R²=0.85).

Figure 7 shows the performance of the pixel subtraction method compared with the H-B scale in subjects with impaired facial nerve function. The correlation coefficient is −0.62 (R²=0.38).

Figure 8 shows the performance of the Nottingham system with the score calculated using coordinates determined by Photoshop compared with the H-B scale. The correlation coefficient is −0.97 (R²=0.94).

Pixel subtraction methods for automation of facial nerve grading is an excellent goal; however, the results have not been ideal. In our study of control subjects, we found high intersubject variability when we compared the linear movement of a point on the face with the area of movement depicted by pixel subtraction. The results suggest that the relationship between the movement of a point on the face and the number of pixels derived using pixel subtraction and thresholding is complicated and highly individual.

Comparing the pixel subtraction method of estimating facial movement with the movement of a single point in the center of the area studied is complicated by several factors. Movement of the point measured with the computer is only in the plane of the x and y axes (measurement of movement in the z axis requires 2 additional cameras, one for each half of the face) and is 2-dimensional, giving rise to what may be an unfair comparison to the 3-dimensional pixel subtraction method. However, the linear relationship we found between the movement of the oral commissure on the x and y axes and the z axis suggests that the z axis component may not be responsible for the marked intersubject variability we found when comparing the pixel area with the movement on the x- and y-axes. Comparing the movement of a single point with the area of change represented by the pixel subtraction method may not be valid because the conformation of the face surrounding point M (Figure 1) may change even after the point M has moved its farthest.

The reliance of pixel subtraction on movement of the entire draped skin surface rather than linear movement of standardized points on the face may cause error in measurement. For example, in the case of extreme laxity of the skin and superficial musculoaponeurotic system we have observed that movement of the half of the face depicted by pixel subtraction may underestimate facial movement. Interestingly, this underestimation may not hold true for grading systems based on linear point motion. Since the measured points (inferior and superior orbit and oral commissure) are near areas of muscular insertion, significant measurable movement of these points still occurs.

Overestimation of facial nerve function by pixel subtraction occurs in 2 situations. First, minute displacement of irregular facial features may falsely amplify actual movement. We observed this in individuals with facial hair and coarsely textured skin. Second, because the normal side of the face pulls the half of the face that is completely paralyzed, actual movement may be overestimated. In many patients with severe paralysis, this point proved to be the most confounding factor since pixel subtraction does not distinguish disorganized movement from organized, directional movement. Figure 9, A through C, which shows an elderly subject, illustrates this point. There is marked pulling of the paralyzed side by the normal side.

Apparent voluntary eye closure is often observed in patients with complete facial paralysis due to tissue elasticity and the pull of gravity on the upper lid. We observed that this phenomenon confounded the pixel subtraction method we used by artificially augmenting the scores of patients with complete facial paralysis. A successful pixel-based system will need to reject this area of the face to avoid this error.

Computer-aided determination of the Nottingham grade offers several advantages over the manual measurement and computation of the grade. Using the computer, the Nottingham score can be calculated at leisure from a permanent image of the patient and measurements can be taken from more than one area on the face at the same time, which is difficult to do with a ruler or calipers.

In this article, we have not compared the accuracy of computer-aided determination with manual determination of the Nottingham grade. Our experience with the 9 subjects with impaired facial nerve function presented herein suggests that the computerized determination of the Nottingham score correlates well with the manually measured score. More data on patients with impaired facial nerve function are being accrued to allow an accurate comparison in a future report.

Photoshop can be used to perform the pixel subtraction analysis described by Neely et al,⁷ as well as the analyses we describe. Although Photoshop approximates the system used by Neely et al, it does not duplicate that system. Nonetheless, currently there are inherent faults with the pixel subtraction method that are eliminated with other grading systems. We believe that the Nottingham system provides a relatively simple, continuous scale. While the computerization we used has the advantage of using commercial software and hardware that are relatively inexpensive, the Nottingham system can be performed without a computer, allowing any practitioner to use it to report results.

Accepted for publication September 12, 1997.

Presented at the Second International Congress on Skull Base Surgery, San Diego, Calif, July 3, 1996.

Reprints: Eric W. Sargent, MD, Department of Otolaryngology–Head and Neck Surgery, St Louis University School of Medicine, 3635 Vista Ave, Box 15250, St Louis, MO 63110-0250 (e-mail: sargente@slu.edu).