The Advanced Dundee Endoscopic Psychomotor Tester.
Mean instrument error contact time enacted by junior and master surgeons on the Advanced Dundee Endoscopic Psychomotor Tester (P = .007). Error bars indicate 2 SEMs.
Mean execution time enacted by junior and master surgeons on the Advanced Dundee Endoscopic Psychomotor Tester (P = .42). Error bars indicate 2 SEMs.
Mean task completion score achieved by junior and master surgeons on the Advanced Dundee Endoscopic Psychomotor Tester (P = .40). Error bars indicate 2 SEMs.
Francis NK, Hanna GB, Cuschieri A. The Performance of Master Surgeons on the Advanced Dundee Endoscopic Psychomotor TesterContrast Validity Study. Arch Surg. 2002;137(7):841-844. doi:10.1001/archsurg.137.7.841
The contrast validity of the Advanced Dundee Endoscopic Psychomotor Tester (ADEPT) was determined by comparing the performance of "master surgeons" with that of surgical trainees (also called junior surgeons) on the system.
Twenty master surgeons and 20 junior surgeons were tested on the ADEPT system. The master surgeons, all of consultant grade, were recruited as established experts of national or international standing in laparoscopic surgery. The junior surgeons were participants of essential laparoscopic courses at the start of their higher surgical training. The ADEPT end points used in the study were instrument error, execution time, and task completion. An analysis of variance was used for the data analysis, with statistical significance set at .05.
Master surgeons incurred a significantly lower instrument error rate than surgical trainees (P = .007), with no significant difference in execution time and the task completion score (P = .42 and P = .40, respectively).
The ADEPT system has contrast validity because master surgeons completed the tasks more accurately without sacrificing execution time.
THE AIM of the selection process for surgical trainees (also called junior surgeons) is to recruit individuals who have the potential (with training) to become safe and competent surgeons. The objective, although self-evident, is not easy to achieve, especially as the process has to cope with changing expectations, needs, and challenges (eg, limited number of training years, regulations on junior physicians' working hours,1 and the increasing demand for surgical specialization). These changes have been accompanied by a restructuring of the surgical training program within the United Kingdom such that it is shorter, incorporates systems that monitor clinical exposure and progress of the surgical trainees, and uses an exit examination. Nevertheless, the realistic expectation that is acknowledged, but rarely stated, by most surgeons involved in surgical training is that surgeons exiting the new system are less experienced and require guidance from their senior colleagues during the initial years of their independent career as consultants or attending surgeons. This is quite acceptable provided these young consultants have all the medical, technical, and humanistic skills necessary for them to mature to competent experienced surgeons in their specialty within a few years of their appointment.
The expected outcomes of these changes that affect surgical health care delivery are contingent on a valid selection process. Regrettably, this has remained unchanged within the United Kingdom since the inception of the National Health Service despite the significant changes in surgical practice, including the establishment of surgical specialty associations and the restructuring of the training program. In essence, the existing selection process lacks standardization and is almost exclusively based on academic achievement or personal knowledge of the candidates and unstructured referees' reports, which often lack objectivity.2 There is an increasing awareness among the surgical colleges and training institutions of the importance of this issue and the need for change. In a recent Delphi-type study involving a cohort of surgeons involved in surgical training from various European countries and the United States, one of the major conclusions reached was the need for improvement of the selection process and for its standardization. The important innate qualities (aptitudes) required for competent surgeons who take good care of their patients were identified by this group, as was the need for the inclusion of aptitude testing in the selection process for surgical trainees.2
There are several contending aptitude-testing systems that merit consideration. The choice of the relevant test for recruitment to a particular profession is crucial and has to be based on the required innate attributes necessary for reaching proficiency on completion of the training program. For recruitment to surgery, computer-controlled psychomotor tests seem to provide a more favorable option than standard psychological tests,3 as each of the latter examines only a single ability. Research using standard psychomotor tests in surgery has not identified all the abilities needed for competent surgical performance. Visuospatial ability is the only ability that seems to correlate with good surgical performance,4 although a recent study5 does not support this observation. Some psychological tests for spatial ability, such as the Embedded Figures Test, are unlikely to be useful for the selection process because of the limited variation between medial subjects on test performance.6 In addition, a chartered psychologist can only administer the battery of psychological tests. This has logistic and financial implications.
In contrast, performance on computer-controlled aptitude assessment systems (CCAASs) provides a global indicator of the level of skills required to execute the tasks set by the system. Studies on the clinical validation of these systems are required to extrapolate the performance on these computer systems with progress in technical operative competence during training and total proficiency to function as an independent competent and safe surgeon on completion of the higher surgical training program or residency. There are 2 approaches to achieve this validation. The first option is to perform a long-term longitudinal study of performance of trainees during and after completion of surgical training. A correlation between performance on the CCAAS and clinical performance during training and subsequently as independent consultant surgeons or attending physicians will reflect the validity of the selection process. This study's design, although ideal, would need to recruit many surgical trainees on a national scale and would require a huge amount of resources over many years, such that it is not feasible.
The second approach is more pragmatic and aims to correlate the performance on the CCAAS with clinical performance in a cross-sectional study. This can be performed by (1) correlating the performance on the CCAAS with the performance of surgical trainees while operating on patients and/or (2) studying the ability of CCAAS testers to distinguish between different levels of technical experience (expert vs junior surgeons).
The Advanced Dundee Endoscopic Psychomotor Tester (ADEPT) is a CCAAS used for the objective evaluation of task performance.7 A pilot study8 on the clinical validity of the ADEPT system has shown that the performance scores on the system correlate with an independent assessment of clinical competence. The ability of the ADEPT system to distinguish between master and junior surgeons was the subject of the present investigation.
The ADEPT system incorporates a dual-gimbal mechanism that accepts 2 standard endoscopic instruments to manipulate various tasks in the target object (Figure 1). The target object used can be varied. The one deployed for the study consisted of a sprung base plate with 5 positioning tasks and a sprung top plate with access holes. The tasks were 2 sliders, a joy stick, a dial, and a toggle switch. Each task entailed manipulation of the top plate with one instrument to enable the other instrument to negotiate the task in the back plate through the access hole. Instructions to the subject are displayed on the computer screen. The system registers instrument error when the instrument is in contact with the sides of the front plate holes.
The ADEPT system uses a 2-dimensional video-endoscopic system (Karl Storz GmbH & Co KG, Tuttlingen, Germany), consisting of a 30° endoscope (model 26033BP, Hopkins II) and a single-chip camera (model 20210030, Telecam). Light within the dome of the ADEPT system is provided by a halogen light source (cold light fountain, 450 V) connected to the endoscope with a fiberoptic light cable (model 495NB), and the image is displayed on a monitor (model PVM-14043MD; Sony Corporation of America, Tokyo, Japan). Two alligator laparoscopic forceps (Karl Storz GmbH & Co KG) are mounted in the gimbal mechanism. For the study, the system was adjusted to provide a 60° manipulation angle (between the 2 instruments), a 30° azimuth angle (between the instrument and the optical axis of the endoscope), and a 60° elevation angle (between the instrument and the horizontal plane). The optical axis-to-target view angle was set at 85°. These angles provide the best ergonomic setup for optimum task performance.9,10 The system software allowed an allocation time of 10 seconds to complete each task and an accuracy tolerance limit of 0.1 second for task positioning.
The end points of the system are instrument error, execution time, and task completion. Task completion involves finishing the task within the allocated time and tolerance limit.
Two groups, each of 20 master or junior surgeons, participated in the study. The master surgeons were recruited from 5 European countries and 2 North American centers; their median age was 47 years (range, 38-56 years). The selection as a master surgeon was based on their peers' standing as excellent technical endoscopic surgeons. The master surgeons performed the ADEPT exercises at the Royal College of Surgeons of Edinburgh, Edinburgh, Scotland.
Junior surgeons were recruited from the participants of the essentials laparoscopic courses at the Surgical Skills Unit, Dundee, Scotland. The participants were surgical trainees either finishing their basic training or at the beginning of higher surgical training (first year). The median age of the junior surgeons was 30.5 years (range, 26-39 years). They performed the ADEPT exercises at the Surgical Skills Unit at the end of their course.
In both groups, each surgeon performed one familiarization run with the ADEPT system and one test run in the same session. A clinical research surgeon (N.K.F.) who is a fellow of the British Psychological Society administered the test.
The data of both groups relating to execution time, contact instrument error time, and task completion score of the ADEPT system were analyzed using an analysis of variance, with statistical significance set at .05.
The data obtained from the study are displayed as box plots (Figure 2, Figure 3, and Figure 4). As a group, the master surgeons committed fewer errors than the surgical trainees, and this difference was significant (P = .007) (Figure 2). This undoubted superior performance by the master surgeons was accompanied by a shorter execution time and a greater task completion score, although these were not significantly different (P = .42 [Figure 3] and P = .40 [Figure 4], respectively). On further analysis, a significant difference in performance was also observed among subjects within each group for all variables (P<.01).
Surgery requires a range of competencies in individuals with certain personality traits.2 The acquisition (with training) of the necessary skills underpinning these competencies is contingent on the presence of a minimum level of innate abilities relevant to these psychomotor skills. In surgery, contrary to other medical professions, a deficiency in these innate abilities (or aptitudes) such that the individual lacks manual dexterity and coordination cannot be compensated by increased cognitive ability. Such an individual may be trained to become an excellent physician, but not a competent operating surgeon. The full range of these aptitudes required for surgery is not known and there is no established method to grade people in their innate psychomotor abilities. What can be detected are the extremes at either end of the norm. The ADEPT system was developed specifically as an objective assessment of psychomotor skills relevant to endoscopic surgery. This technology is able to measure the level of innate manual dexterity, aiming, and eye-hand coordination, thereby identifying individuals who fall sufficiently below the norm as to be unsuitable for a surgical career. The system provides a standard method of test application, as all subjects are assessed in the nature of the test and the instructions displayed on the computer screen. The face validity of the system includes using a real endoscopic imaging system and standard laparoscopic instruments. In addition, the movement of instruments in the gimbal mechanism of the system has the same degrees of freedom as endoscopic instruments through access ports. Scoring the performance by the computer software eliminates interassessor variability. The reliability of the system has also been proved in internal consistency and test-retest reproducibility.11
The clinical validity of the ADEPT system has also been studied in one pilot study on surgical trainees. This showed that ADEPT performance scores correlated with a blind independent assessment of clinical and technical competence.9 In addition, the ability of the ADEPT system to discriminate between subjects with the same level of clinical experience has been demonstrated.11 The present study has confirmed the contrast validity of the system because it was clearly able to distinguish between experts and trainees. As a group, the master surgeons completed the ADEPT tasks with a significantly lower error rate and a shorter execution time than the surgical trainees, who were on average 25 years younger. In addition, the ADEPT system identified significant differences in the performance levels between individuals of either group.
The results of the present study confirm the potential role of the ADEPT system in the selection of surgical trainees if assessment of psychomotor dexterity is to be included in the selection process in the future. In this respect, the generic nature of the ADEPT system should be stressed, as the system can accommodate different task plates of varying complexity (mechanical, polymeric, and animal tissue) and cover a range of tasks from simple "pick and place or move" exercises to complex ones (eg, suturing and knot tying). The ADEPT system can also be used to identify surgical trainees who require additional training to improve their operating skills before being allowed to operate on patients, and this training may be provided on the system itself. However, longitudinal studies are required to confirm this.
This study was supported by Santi Corsaro, MD, Ethicon Endo-Surgery, Inc, Cincinnati, Ohio.
We thank the master and junior surgeons who participated in the study.
Corresponding author and reprints: Alfred Cuschieri, MD, FRSE, Department of Surgery and Molecular Oncology, Ninewells Hospital & Medical School, University of Dundee, Dundee DD1 9SY, Scotland (e-mail: firstname.lastname@example.org).