Channel 1 includes electrodes 1 and 2 (inferolateral quadrant). Channel 2 includes electrodes 2 and 3 (superolateral quadrant). Channel 3 includes electrodes 4 and 5 (superomedial quadrant). Channel 4 includes electrodes 5 and 6 (inferomedial quadrant).
Shown is an extract of the EMG signal related to a blink, with definition of the 4 thresholds used by the real-time blink detection algorithm.
Shown is the flow of the real-time algorithm to detect blinking from surface electromyography (EMG).
Four blink peaks are presented for each of the 4 facial activities recorded in the experiments. Dots indicate the detected blinks.
Shown are the median (interquartile range) F1 scores obtained for each electromyographic channel and the 4 facial activities tested and the overall value.aIndicates a statistically significant difference (P < .05) between channels.
Frigerio A, Cavallari P, Frigeni M, Pedrocchi A, Sarasola A, Ferrante S. Surface Electromyographic Mapping of the Orbicularis Oculi Muscle for Real-Time Blink Detection. JAMA Facial Plast Surg. 2014;16(5):335-342. doi:10.1001/jamafacial.2014.283
Facial paralysis is a life-altering condition that significantly impairs function, appearance, and communication. Facial rehabilitation via closed-loop pacing represents a potential but as yet theoretical approach to reanimation. A first critical step toward closed-loop facial pacing in cases of unilateral paralysis is the detection of healthy movements to use as a trigger to prosthetically elicit automatic artificial movements on the contralateral side of the face.
To test and to maximize the performance of an electromyography (EMG)-based blink detection system for applications in closed-loop facial pacing.
Design, Setting, and Participants
Blinking was detected across the periocular region by means of multichannel surface EMG at an academic neuroengineering and medical robotics laboratory among 15 healthy volunteers.
Main Outcomes and Measures
Real-time blink detection was accomplished by mapping the surface of the orbicularis oculi muscle on one side of the face with a multichannel surface EMG. The biosignal from each channel was independently processed; custom software registered a blink when an amplitude-based or slope-based suprathreshold activity was detected. The experiments were performed when participants were relaxed and during the production of particular orofacial movements. An F1 score metric was used to analyze software performance in detecting blinks.
The maximal software performance was achieved when a blink was recorded from the superomedial orbit quadrant. At this recording location, the median F1 scores were 0.89 during spontaneous blinking, 0.82 when chewing gum, 0.80 when raising the eyebrows, and 0.70 when smiling. The overall performance of blink detection was significantly better at the superomedial quadrant (F1 score, 0.75) than at the traditionally used inferolateral quadrant (F1 score, 0.40) (P < .05).
Conclusions and Relevance
Electromyographic recording represents an accurate tool to detect spontaneous blinks as part of closed-loop facial pacing systems. The early detection of blink activity may allow real-time pacing via rapid triggering of contralateral muscles. Moreover, an EMG detection system can be integrated in external devices and in implanted neuroprostheses. A potential downside to this approach involves cross talk from adjacent muscles, which can be notably reduced by recording from the superomedial quadrant of the orbicularis oculi muscle and by applying proper signal processing.
Level of Evidence
The loss of blinking represents a debilitating feature of peripheral facial paralysis, and meticulous eye care is a cornerstone of proper management. Blink rehabilitation via pacing and robotic technology has great but as yet unmet potential. Closed-loop facial pacing may be a possible solution toward reanimating symmetrical blink and other facial movements in individuals with unilateral peripheral facial paralysis.1- 7 Because spontaneous blinks are normally symmetrical, in unilateral paralysis cases a biomimetic device can record a biosignal when the healthy eye blinks spontaneously; the signal can be further processed to prosthetically assist blinks on the paralyzed side. Previous studies have shown the feasibility of facial pacing in animal models1,4- 10 and in humans.2,3,11
This work focuses on the recording arm of the loop (ie, detecting blinks). Tracking an infrared light beam passing across the eye surface has been shown to provide a simple, noninvasive indication of eyelid closure in humans.12 Another valuable technology for movement detection is electromyography (EMG). The electrical activity of a muscle starts before the production of force occurs and can be detected by means of EMG recording before the onset of movement. This property makes EMG a potentially useful biosignal for applications in closed-loop facial pacing.1- 4,13 Moreover, an EMG recording apparatus can be easily integrated in external devices and in implanted neuroprostheses.
The orbicularis oculi muscle is one of the superficial muscles of facial expression. Invested by the superficial musculoaponeurotic system, muscle contracture is translated into movement of the overlying tissues by the fibrous septae extending from the superficial musculoaponeurotic system into the dermis. The muscle may be arbitrarily divided into the orbital and palpebral parts, with the latter being further divided into the preseptal and pretarsal portions. The palpebral part is used in blinking and in voluntary winking, while the orbital part is used in forced closure. Individual myofibers of the orbicularis oculi muscle are short, end intrafascicularly, and are of heterogeneous lengths varying regionally within the muscle, with most of the neuromuscular junctions in the medial and lateral canthal regions of the preseptal portion of the lid.14
Preliminary studies1- 4 have explored blink detection by means of surface EMG recording of the orbicularis oculi muscle. Electrical noise from adjacent muscles represents a potential downside of surface EMG–based blink detection for facial pacing. Our team has previously attempted to address this issue by using standard surface EMG electrodes in a multichannel recording fashion: 2 electrodes were placed on the orbicularis oculi muscle surface, and another channel recorded the masseter and zygomatic muscles.13 Custom software was designed to prevent stimulation triggers when the zygomatic muscle, the masseter muscle, or both were active.13
The aims of the present study were to seek the best recording location across the periocular surface for the purpose of the envisioned application and to reduce the number of EMG channels to be used to one. Surface EMG mapping the orbicularis oculi muscle was performed in healthy individuals, and blink detection at different recording locations was analyzed in relation to blinking and other common facial movements.
The research protocol was approved by the ethics committee of the University of Milan Medical School, Milan, Italy, in accord with the standards of the 1964 Declaration of Helsinki. The experiments were performed between December 20, 2012, and February 28, 2013, among 15 healthy adult volunteers (8 women and 7 men) with a mean (SD) age of 24.13 (2.99) years. Ineligibility of individuals meeting any of the exclusion criteria (ie, any ophthalmological abnormality or a clinical history of neuromuscular disease) was determined. All participants were fully informed about the procedure and provided oral consent; enrolled individuals received no compensation for their participation in the study.
Participants were committed to one research session lasting approximately 20 minutes. This included informed consent, setup time, and cleanup time.
After skin preparation with alcohol wipes of the frontal and periocular regions on one side of the face, 6 surface monopolar microelectrodes (with a 2-mm recording surface diameter) were coated with conductive gel and taped epicutaneously corresponding to the following fixed anatomical landmarks: (1) inferolateral orbital rim above the frontal process of the malar bone, (2) lateral canthal tendon, (3) superolateral orbital rim above the frontomalar suture, (4) superomedial orbital rim above the frontonasal suture, (5) medial canthal tendon, and (6) inferomedial orbital rim above the frontal process of the maxilla.
A ground pregelled electrode (with a 24-mm diameter) was taped on the skin of the forehead. Figure 1 shows the multichannel recording of the orbicularis oculi muscle in relation to fixed anatomical landmarks (ie, ligaments and bony structures).
Four recording channels were obtained from the 6 monopolar electrodes. For each recording channel, the intercenter electrode distance was approximately 1.2 cm. A PC running a software program (Scilab/Scicos; Scilab Enterprises) on a notebook (Linux-based operating system; Linux Foundation) with a real-time kernel (RealTime Application Interface for Linux) was used to obtain the EMG signals and a synchronization signal that changed its value as the video was started. This open-source platform includes a specific module for the stimulation arm of the envisioned closed loop. The EMG signals were obtained at 1024 Hz by means of a commercial polygraph (Porti; TMS International BV). The polygraph uses bipolar amplifier technology with a low amplification gain of 20 and a high-resolution analog to digital conversion of 22 bits. It does not include a filtering stage. The polygraph amplifies the input signal against the mean reference of the incoming signals (ie, the common-mode signal). The common-mode range was −2 to 2 V, and the common-mode rejection ratio was 100 dB or higher. Real-time data transmission to the PC is achieved via a USB connection. A gel electrode with low impedance was placed as the ground, and shielded cables were used for all electrodes. The synchronized video camera (D7000; Nikon) recorded the participant’s face across the whole session and was then used to facilitate off-line data processing necessary to classify the blinks detected by the EMG processing software.
Testing involved asking individuals to sit and look straight forward for approximately 5 minutes while watching a video file that was played on a tablet. Participants were asked to watch a narration for 1 minute while maintaining a neutral head position. They were then asked to perform a sequence of 3 facial activities as directed by the video narrator in the order of smiling, raising the eyebrows, and chewing. A narration at the start of each expression described what was being performed. Participants were asked to hold each expression for a few seconds. Between each expression was a neutral expression when they could relax and look straight ahead without holding any particular expression.
During the airing of the video file, a calibration phase was followed by a testing phase. Sessions corresponding to different facial activities lasted 20 seconds each during calibration and 45 seconds each during testing.
Custom software processed the EMG signals to detect blinks in real-time and to trigger an output representing the stimulation arm of the envisioned closed-loop system. The algorithm detected blink events before the movement artifact (Figure 2) to process the signal in time to elicit a synchronous movement. The algorithm was based on 4 thresholds set for each participant during the calibration phase. The 4 analyzed facial activities were included in the calibration to consider intrasubject variability of the EMG signal for each activity.
During calibration, a peak detection–based algorithm was used to determine blinking peak values. The central 100 samples of each interblink interval (ie, the interval between 2 consecutive blink peaks) were used to compute the maximal interblink baseline value. A Monte Carlo simulation with a trial and testing approach over the calibration data set of the 4 channels allowed definition of the thresholding parameters. The robustness of these parameter definitions was also tested over a set of preliminary acquisitions. The amplitude threshold was defined as 65% of the mean of all the detected blink peak values. Similarly, the baseline threshold was defined as 2.5 times the mean of all the registered maximal values in the interblink phases.
In addition to these 2 thresholds, slope and variability thresholds were computed to take into account the dynamic behavior of blinking and to discard movement artifacts obtained during other facial movements, normally slower than spontaneous blinking. The slope-based threshold was determined by first computing the slope between the first sample above the baseline threshold and the following samples greater than the first. The maximal slope was then computed for each window of 40 samples. Finally, the threshold was set at the mean value of all of them. To obtain the last threshold, signal variability was computed as the mean difference between every 2 consecutive samples above the baseline threshold. The mean variability value was first calculated for each window of 40 samples, and the threshold was then set at the maximal value of all of them.
The flow diagram in Figure 3 shows the real-time algorithm in detail. The EMG filtering, applied to windows of 40 samples, consisted of a cascade of a second-order notch filter and an eighth-order low-pass filter (with a cutoff frequency of 20 Hz).
Once filtered, the resulting signal is compared with the baseline threshold; when the signal overcomes the baseline threshold, the slope and variability are computed and compared with the corresponding thresholds. Finally, when both of the latter thresholds are overcome and the signal is greater than the amplitude threshold, a blink event is detected, and stimulation is triggered.
After every blink detection, stimulation triggers are prevented for an interval correlating with the mean human blink duration of 300 milliseconds.15 This lockout period avoids any EMG feed-through effect from the stimulated side of the face to the other, which could lead to an undesired positive feedback loop in a final neuroprosthesis application.
Off-line analysis was performed by comparing videos of the experiments with the EMG blink detection circuit output. A blink was noted when the upper and lower edges of the pupil were at least partially covered by the eyelids. Each video was viewed at least 3 times to ensure that all blinks were noted, and observers (M.F. and A.S.) were blinded to the EMG blink detection circuit output.
By comparing the video-recorded blinks with the output of the software, we designed a software program to analyze the correctly and incorrectly recognized blinks for each individual and EMG channel by means of the numbers of true positives and false positives (TPs and FPs) and true negatives and false negatives (TNs and FNs). For TPs, the system detected a blink event that truly occurred. For FPs, the system detected a blink event, but the participant did not blink. For TNs, the participant did not blink, and the system did not detect any blink. For FNs, the participant blinked, and the system failed to detect the event.
The large number of TNs compared with other parameters resulted in an imbalanced analysis data set, and the accuracy of our system could have been spuriously high. Therefore, we computed precision and recall metrics, as well as the harmonic mean of them (called the F1 score), because of their widespread use and suitability for this type of classification analysis.16
The precision (P), recall (R), and F1 score values were computed from TP, FP, and FN parameters as presented in the following equations:
P = TP / (TP + FP),
R = TP / (TP + FN), and
F1 Score = 2 * P * R / (P + R).
After verifying that the F1 scores obtained were not normally distributed, Kruskal-Wallis test was applied to compare the results from different EMG channels. Dunn-Sidak post hoc test was performed to determine which pairs of EMG channels were significantly different. P < .05 was considered statistically significant. Analysis was performed using a statistical toolbox (MATLAB; The MathWorks, Inc).
The experiments were well tolerated by the participants, with no reported discomfort. Figure 4 shows the signals obtained from the superomedial corner of the orbit (channel 3); the 4 segments correspond to the 4 facial activities. During spontaneous blinking, the EMG signal was characterized by a less noisy baseline compared with the other facial movements. The movement artifact had a similar amplitude under all conditions.
The mean latency between the EMG onset of the recorded orbicularis oculi muscle and the stimulation output was approximately 22 milliseconds. This is in full compliance with specific time constraints for synchronous movements (33 milliseconds).
The Table lists the precision, recall, and F1 score values of the EMG recordings at 4 different locations. For each channel, the median (interquartile range) obtained among participants is reported for the 4 facial activities. In addition, for each channel an overall value for the precision, recall, and F1 score values was computed for all the facial activities together. The best performance was obtained by channel 3 (the superomedial quadrant), for which the median F1 score was always 0.70 or higher.
Statistical analysis was performed for each of the facial movements and for the overall values, and the median (interquartile) results are shown in Figure 5. Kruskal-Wallis test showed a significant difference across channels (P < .05) during spontaneous blinking and for the overall value.
Despite much effort by investigators to restore eyelid movement and smile in patients with facial paralysis, most agree that the results of facial reanimation have significant room for improvement. Extensive reconstruction procedures have been described for blink restoration.16 Closed-loop facial pacing represents a potentially innovative solution for the rehabilitation of facial movements in cases of unilateral facial paralysis, and several studies2,3,11 have explored this application as proof of concept in humans. To date, research efforts by our group have been directed toward the prosthetic rehabilitation of blinking,3,13 the loss of which is a life-altering condition for most affected individuals.
We envision a closed-loop blink-pacing device, either external or implantable, that integrates a blink detection apparatus, a microcontroller for signal processing, and an activator. The recording arm of the loop is a crucial point in the design of the described neural interface and was the target of this research.
A blink detection algorithm for closed-loop pacing has to meet several critical goals. These are discussed below and include (1) enabling real-time detection, (2) ensuring the lowest possible rate of nonblink activities generating false trigger events, and (3) overcoming interindividual and intraindividual variability.
The overall timing of the apparatus (recording, processing, and stimulation) is critical to elicit synchronous facial movements. Normal eyelid movements in humans are highly conjugated, and the side-to-side threshold for detecting eyelid asymmetry is lower for blinking than for other movements.17 Human visual information updates at a rate of approximately 30 to 33 Hz.18 Recent evidence demonstrated that the time-delay detection threshold for blinking is approximately 33 milliseconds and that it is the shortest among common facial movements (smiling, raising the eyebrows, and blinking).19 Therefore, the ideal delay in closed-loop blink pacing between the onset of blinking on the healthy and paralyzed sides should be no longer than 33 milliseconds.
The early detection properties of EMG recording allow the gain of a few milliseconds. Indeed, regardless of their reflexive, voluntary, or spontaneous origin, all blink movements start 10 to 12 milliseconds after the onset of orbicularis oculi muscle EMG activity.20 In addition, a delay of approximately 10 milliseconds from the stimulation output to the paralyzed eyelid contraction must be taken into account.21 Based on these tight time constraints, a threshold-based algorithm was preferred. The developed software complied well with this requirement: the stimulation output was triggered with a mean latency of approximately 22 milliseconds from the beginning of the healthy orbicularis oculi muscle activity.
Preventing false trigger events has been examined by our research team in relation to electrical cross talk from adjacent muscles.13 The aim of the present study was to test the performance of surface EMG–based blink detection across the periocular region by mapping the surface of the orbicularis oculi muscle. The orbital part of the orbicularis oculi muscle extends in a wide circular fashion around the orbit, interdigitating with other muscles of facial expression. It has a curved origin from the medial orbital margin, being attached to the superomedial orbital margin, maxillary process of the frontal bone, medial canthal tendon, frontal process of the maxilla, and inferomedial orbital margin. These fixed landmarks were chosen as recording locations for our mapping study. Despite the fact that they are placed above the orbital part of the orbicularis oculi muscle, electrodes can reliably detect the electrical activity of the palpebral part of the lids during spontaneous blink activity. The results showed that the superomedial quadrant of the orbicularis oculi muscle orbit represents a superior recording location compared with the traditionally used inferolateral quadrant: post hoc analysis showed that, in both cases, the superomedial quadrant (channel 3) significantly outperformed the commonly used inferolateral quadrant (channel 1).13 The EMG recording from this location performed better than recording from other quadrants in terms of precision and recall. This result represents a considerable improvement in the blink-detection apparatus by avoiding the need for multichannel recording to distinguish blinks from other facial movements.
In addition to FP stimulus generation from cross talk from adjacent muscles, stimulation artifacts can generate false trigger events.4 However, our software eliminates these events by the use of a blanking period of 300 milliseconds after any stimulation is delivered.
The interindividual and intraindividual variability of the surface EMG signal may affect the accuracy of blink detection. Indeed, the algorithm may be able to adapt to each person and to each electrode replacement. Our algorithm used 4 different thresholds that accounted for signal changes in amplitude, as well as the various signal variability and steepness across different facial activities.
A possible limitation of this study is that the stability of the custom software for longer acquisitions was not investigated. For long acquisitions, the quality of the contact between the electrode and the skin can change and may alter the EMG signal. An update of the thresholds would be required to prevent this occurrence.
To date, possible options to record spontaneous eye blink activity have been explored. Moving forward, blink-elicitation studies in patients with unilateral palsy are necessary to complete the envisioned pacing loop.
Accepted for Publication: March 28, 2014.
Corresponding Author: Alice Frigerio, MD, PhD, Facial Nerve Center, Carolyn and Peter Lynch Center for Laser and Reconstructive Surgery, Division of Facial Plastic and Reconstructive Surgery, Department of Otology and Laryngology, Massachusetts Eye and Ear Infirmary, Harvard Medical School, 243 Charles St, Boston, MA 02114 (firstname.lastname@example.org).
Published Online: July 17, 2014. doi:10.1001/jamafacial.2014.283.
Author Contributions: Dr Frigerio and Ms Sarasola contributed equally to this work and had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Frigerio, Cavallari, Sarasola.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Frigerio, Cavallari, Ferrante.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Frigeni, Sarasola.
Obtained funding: Cavallari, Pedrocchi.
Administrative, technical, or material support: Frigerio, Pedrocchi, Sarasola.
Study supervision: Frigerio, Cavallari, Pedrocchi, Ferrante.
Conflict of Interest Disclosures: None reported.
Previous Presentations: This study was presented in part at the Combined Otolaryngology Spring Meetings; April 10-14, 2013; Orlando, Florida; at the 12th International Facial Nerve Symposium; June 28 to July 2, 2013; Boston, Massachusetts; and at the 18th International Functional Electrical Stimulation Society Annual Conference; June 6-8, 2013; San Sebastian, Spain.