Each participant completed a questionnaire for each video viewed (2 sets of 25 videos). PGY indicates postgraduate year.
Customize your JAMA Network experience by selecting one or more topics from the list below.
Russell KA, Brook CD, Platt MP, Grillone GA, Aliphas A, Noordzij JP. The Benefits and Limitations of Targeted Training in Flexible Transnasal Laryngoscopy Diagnosis. JAMA Otolaryngol Head Neck Surg. 2017;143(7):707–711. doi:10.1001/jamaoto.2017.0120
Can targeted laryngoscopy teaching improve diagnostic abilities in de novo learners?
In this prospective study of fiberoptic laryngoscopy interpretations in interns and attending physicians, significant improvement occurred after targeted teaching about vocal cord abnormalities.
Learning of flexible laryngoscopy can be improved with the use of a teaching video.
Targeted laryngoscopy training can be used successfully in de novo learners.
To determine the value of targeted laryngoscopy education in interns.
Design, Setting, and Participants
This prospective study of fiberoptic laryngoscopy interpretations enrolled 13 participants in an academic hospital setting from August 1 to December 31, 2015. Participants included 10 postgraduate year 1 emergency and otolaryngology interns and 3 board-certified otolaryngology attending physicians.
Participants viewed 25 selected and digitally recorded fiberoptic laryngoscopies and were asked to rate 13 items relating to abnormalities in the pharynx, hypopharynx, larynx, and subglottis; the level of concern; and confidence with the diagnosis. A laryngoscopy teaching video was then administered to the interns before rating a second set of 25 videos. Improvement in diagnosis and intraclass correlation coefficients (ICC) were calculated for each question and compared between the first and second administration.
Main Outcomes and Measures
Improvement in correct diagnosis of abnormalities in recorded laryngoscopies.
All 13 participants completed the interventions. The ICCs for all questions were generally low for the intern groups and higher for the attending group. For vocal cord mobility, a preintervention ICC of 0.25 (95% CI, 0.16-0.37) improved to 0.47 (95% CI, 0.36-0.59) among interns after the intervention. The ICCs for vocal cord mobility were higher among attendings for the preintervention (0.89; 95% CI, 0.84-0.93) and postintervention (0.89; 95% CI, 0.83-0.93) assessments. Minimal improvement was observed in intern scores for base of tongue abnormalities, subglottic stenosis, vocal cord abnormalities, level of comfort, level of concern, pharyngeal abnormalities, or laryngeal, pharyngeal, and hypopharyngeal masses.
Conclusions and Relevance
Learning of flexible laryngoscopy can be improved with the use of a teaching video; however, additional interventions are needed to attain competence in accurately diagnosing upper airway lesions. Clinicians who seek to perform flexible laryngoscopy require robust training.
Flexible transnasal laryngoscopy (FTNL) is a commonly used diagnostic procedure for evaluation of the upper aerodigestive tract. Findings on these procedures can range from normal to minor abnormalities to critical diagnoses that necessitate immediate intervention. No standardized method for teaching this important diagnostic skill currently exists, and trainees learn by repetition of the procedure on patients in the inpatient and outpatient settings. In a previous study,1 we concluded that otolaryngology residents reached competency in FTNL compared with otolaryngology attendings by postgraduate year (PGY) 3. By that time, the PGY-3 residents had undergone approximately 15 months of dedicated otolaryngology training and had performed numerous FTNLs without the administration of any targeted training or additional educational tools for the procedure. The purpose of this study was to determine whether administering an educational tool in the form of an educational narrated video could improve novice clinicians’ ability to diagnose various upper aerodigestive diagnoses and confidence in identifying worrisome airways using FTNL.
This prospective study of fiberoptic FTNL interpretations used archived endoscopy videos taken during routine clinical evaluations in the otolaryngology outpatient clinic of Boston University Medical Center, Boston, Massachusetts, from January 1, 2006, through December 31, 2013. This study was approved by the institutional review board at Boston University Medical Center, which waived the need for informed consent.
We created 2 sets of 25 videos. Participants were asked to view 1 set of 25 videos, followed by a teaching video, and then asked to view a second set of 25 videos. A total of 50 videos were administered to all participants in 2 groups of 25 videos. Of the videos in each set of 25, 21 were unique and 4 were repeated to calculate intrarater reliability.
A standardized questionnaire was given to each participant immediately after each video that they viewed, totaling 2 sets of 25 questionnaires for each participant (Figure). We designed the questionnaire to be general and to address each subsite in the larynx to reduce priming. Participants were aware of the possibility of viewing normal and abnormal findings. Approximately 30 minutes elapsed between viewing the first set of videos and the second set of videos. Participants were asked to rate 13 items relating to abnormalities in the pharynx, hypopharynx, larynx, and subglottis; the level of concern; and confidence with the diagnosis, with the level of abnormalities or concern based on an attending otolaryngology rating (ie, criterion standard). All videos excluding the narrated instructional video were administered without sound to eliminate bias by recognition of the examiner’s voice (eg, a specific otolaryngologist who specializes in cancer).
Participants for the study were chosen to represent de novo learners of fiberoptic FTNL based on having limited to no previous training in the area but motivation to improve their current diagnostic skills. Emergency department (ED) and otolaryngology interns were given the option to participate in the study. Seven ED and 3 otolaryngology interns participated for a total of 10 interns. Three attending otolaryngologists at our institution also rated each set of videos. The mean of these scores was calculated and used as the criterion standard. Of the attending otolaryngologists, 1 was a fellowship-trained rhinologist and 2 were general otolaryngologists. The PGY1 interns were administered the instructional video between the first and second sets of videos, and the attending otolaryngologists were not administered the teaching video. The information from the questionnaires was then recorded in a spreadsheet for analysis.
All endoscopies were performed on the same type of digital video laryngoscopes (type ENF-V2; Olympus) and accessed from digital archives. All endoscopies used in the study were of adult patients. We specifically selected the videos from the archives for the presence of certain abnormalities or findings. The videos represented roughly the same number of normal and distinct pathologic conditions in each set of 25. Pathologic conditions represented in videos included vocal cord paralysis, subglottic stenosis, and laryngeal and pharyngeal masses. The videos were deidentified and edited to representative examinations of 20 to 50 seconds by one of us who is a fellowship-trained laryngologist (J.P.N.). None of us who participated in the selection or editing process reviewed the examinations as participants. Videos included selections from the nasopharynx, oropharynx, hypopharynx, and larynx. None of the endoscopies passed beyond the glottis, although the subglottis was represented in some of the videos. The purpose of editing was to eliminate nondiagnostic portions of the examination at the beginning and the end of the videos that could bias the results. The videos were contiguous examinations except for 1 video that had a long, redundant segment removed. Videos were labeled and sorted in 2 folders with each having 4 videos repeated to determine intrarater reliability. A narrated, educational, 8-minute 17-second video was created using selections from the first set of 21 videos. The narration was coauthored and approved by one of us (J.P.N.) to discuss relevant anatomy, pathologic changes present, and the level of concern for that airway.
Videos were administered from August 1 to December 31, 2015. Data analysis was performed using Excel with the Real-Statistics Resource Pack (Microsoft Corp).2 Interrater reliability was assessed with intraclass correlation coefficients (ICCs) for all interns compared with attendings, ED interns compared with attendings, otolaryngology interns compared with attendings, and among the attendings. In addition, interrater ICCs were calculated for pooled vocal cord mobility (left and right vocal cord mobility) because we believed that this represented a single diagnostic skill rather than 2 independent areas of observation.
We generated ICCs to compare the degree of agreement within each training group (ED and otolaryngology interns and attending otolaryngologists) and the degree of agreement between each group of interns and the attending otolaryngologist criterion standard. A total of 13 participants were included. For interpretation, we used reference ICCs of 0.80 to 1.00 for very good agreement, 0.60 to 0.79 for good agreement, 0.40 to 0.59 for moderate agreement, 0.20 to 0.39 for fair agreement, and 0 to 0.19 for poor agreement.
The ICCs for all questions before and after the intervention were generally low for the interns of both groups and higher for the attending group (Table 1 and Table 2). The ICCs for ED interns compared with attendings were generally slightly lower than for the otolaryngology interns. Improvement was noted between the prevideo and postvideo pooled right and left vocal cord mobility scores. The ED interns’ preintervention ICC was 0.25 (95% CI, 0.16-0.39), and the postintervention ICC was 0.44 (95% CI,0.33-0.57). The otolaryngology interns’ preintervention ICC was 0.34 (95% CI, 0.08-0.56), and the postintervention ICC was 0.50 (95% CI, 0.26-0.68). These findings moved both groups from a preintervention agreement of fair to moderate agreement, suggesting a small but real improvement in diagnostic ability for vocal cord mobility. We found minimal improvement in intern scores for base of tongue abnormalities, subglottic stenosis, vocal cord abnormalities, level of comfort, level of concern, pharyngeal abnormalities, or laryngeal, pharyngeal, and hypopharyngeal masses.
Finally, when assessing ICCs after the second viewing of the FTNL videos, we found that all raters demonstrated good to very good agreement. For ED interns, ICCs were 0.78 (95% CI, 0.74-0.81) before and 0.81 (95% CI, 0.77-0.84) after the intervention. For otolaryngology interns, ICCs were 0.83 (95% CI, 0.76-0.88) before and 0.89 (95% CI, 0.84-0.92) after the intervention. For attendings, ICCs were 0.93 (95% CI, 0.91-0.95) before and 0.95 (95% CI, 0.93-0.96) after the intervention.
The use of educational tools, including videos, simulation exercises, and online modules, has been increasing for medical education in recent years.3 Many medical specialties, including otolaryngology, general surgery, obstetrics and gynecology, anesthesiology, pediatrics, and emergency medicine, have shown improvements in accuracy and skill and decreases in procedure times with the use of these educational tools.4-7 Previous studies using simulation exercises for fiberoptic FTNL have shown improved skills in trainees performing the procedure on patients, allowing them to focus on other aspects of the examination.4 The relatively new restrictions on resident work hours necessitate more efficient training for surgical residents because of concern for decreased numbers of operative cases by residents after the implementation of the 80-hour work week.8-10 Educational tools designed and administered appropriately may help a resident achieve competency faster, leaving time for other, more time-intensive learning opportunities, such as working in the operating room.
Many additional examples of using simulation or educational tools for resident learning have had positive results. Allak et al11 developed a simulator for performing rigid esophagoscopy. Otolaryngology residents with varied levels of experience showed improved completion of procedural steps, higher Objective Structured Assessment of Technical Skills scores, and decreased force applied to tissue by the esophagoscope after the simulation exercise. Patel et al12 used simulation exercises that included a didactic lecture, procedural video, and simulation of laparoscopic salpingectomy on a porcine cadaver performed by the intervention group compared with a control group who underwent traditional training per routine residency rotations. The intervention group demonstrated improvement in surgical technique Objective Structured Assessment of Technical Skills scores compared with the traditional training group. First-year obstetric residents learned cesarean deliveries through an online multimedia module as detailed in a study by York et al.13 Abdelsattar et al14 had residents and fellows view a set of selected surgical procedure videos offering points for identifying a checklist of items, including visible anatomy, visualizing poor surgical technique, and correctly identifying surgical instruments on the screen. They believed that this tool was a way to identify trainees ready for hands-on training and those who may require additional preemptive education. Other studies have also found that the administration of video tools enhances surgical resident education and skill.13,15
Prior studies16 have also shown that trainees find videos to be useful educational tools. These videos can be designed to be short, specific, and easily accessed to provide succinct, quick, and useful educational material. Levitt et al17 demonstrated that ED attending physicians’ misinterpretation rate of cranial computed tomographic scans was decreased by 4% after a 1-hour educational session on interpreting these types of scans. Rowse et al18 discussed how the current generation of learners, including residents, favor the use of technology to obtain new information. They incorporated that idea into a 90-second video clip administered to residents demonstrating proper fine-needle aspiration biopsy techniques. The viewing of this video increased their residents’ performances of this skill when viewed within 2 weeks before evaluation.
Our data suggest that a short, narrated educational video can be useful in accelerating learning. Overall, the ICCs for the intern group were low, as expected given their lack of training. Improvement was noted between preintervention and postintervention ICCs in regard to pooled right and left vocal fold mobility (overall interns improved from 0.25 [95% CI, 0.16-0.37] to 0.47 [95% CI, 0.36-0.59]), although these results were still lower than attendings’ ICCs for vocal cord immobility (0.89 [95% CI, 0.83-0.93]). Although this specific diagnostic area improved, the intervention group did not reach the diagnostic ability of otolaryngology attendings with many years of experience, and they showed minimal improvement in other areas of the examination. As we have previously shown,1 reliability is poorer in some aspects of FTNL despite completion of training, but this finding also suggests that a simple training video is not a substitute for years of clinical experience.
Clinicians with limited FTNL experience could benefit from viewing educational videos such as this one. An ED physician who does not perform many flexible laryngoscopies or has extended periods between performances could quickly watch this type of video before performing FTNL. The video would provide a quick refresher on common aerodigestive abnormalities for the clinician to keep in mind during his or her examination. The video could also assist the clinician in determining whether the patient has a concerning airway that may necessitate an emergent intervention vs referring for an outpatient evaluation by an otolaryngologist with more experience.
The possibility of expanding the video tool could lead to further improvement. Learning tools focusing on specific upper aerodigestive subsite abnormalities that are rare or more difficult for the novice to identify could increase the trainees’ ability to recognize abnormalities in those areas and is an area open to further research. Additional studies involving emergency medicine attendings and multiple institutions may be useful for further investigation.
Limitations of the study include a small sample size and short interval between the participants’ viewing of the first and second sets of videos. A longer interval between administering the sets of videos could help to further investigate learning of the participants rather than risking the possibility of measuring simple recall, although the videos administered in the preintervention and postintervention sessions were different. The authors were surprised, however, at the limited improvement seen overall in ICCs, even with the short interval between administrations. The ICCs for level of comfort with diagnosis by participants were low and even decreased after intervention. This finding could reflect an overall new appreciation for the vast complexity of the upper airway and endoscopic diagnosis with which the trainees were previously unfamiliar. It suggests that a relatively simple training tool, such as the video, does not replace the experience acquired through mentorship and guided clinical learning. Some of these early trainees may also lack the foundational knowledge to assess laryngeal anatomy at a level high enough to recognize normal and abnormal, and they may have benefitted from more extensive formal didactic training on upper airway anatomy.
Diagnostic interpretation of FTNL can be improved with the use of educational tools, including a teaching video of laryngoscopies, especially for vocal cord mobility. Additional experience, however, is needed to attain competence in accurately diagnosing upper airway lesions. This tool is not intended to replace appropriate supervision of trainees by experienced clinicians in the area of flexible laryngoscopy. Trainees and clinicians with limited experience in FTNL may benefit from a tool like the one presented herein in the appropriate setting.
Corresponding Author: Michael P. Platt, MD, MSc, Department of Otolaryngology–Head and Neck Surgery, Boston University Medical Center, 820 Harrison Ave, FGH Building, 4th Floor, Boston, MA 02118 (email@example.com).
Accepted for Publication: January 14, 2017.
Published Online: May 4, 2017. doi:10.1001/jamaoto.2017.0120
Author Contributions: Dr Brook had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Russell, Brook, Platt, Noordzij.
Acquisition, analysis, or interpretation of data: Russell, Brook, Platt, Grillone, Aliphas.
Drafting of the manuscript: Russell, Brook, Platt.
Critical revision of the manuscript for important intellectual content: Brook, Platt, Grillone, Aliphas, Noordzij.
Statistical analysis: Brook.
Administrative, technical, or material support: Russell, Platt, Aliphas.
Study supervision: Brook, Platt, Grillone, Noordzij.
Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported.
Create a personal account or sign in to: