Failure of a Web-Based Educational Tool to Improve Residents' Scores on the American Board of Surgery In-Training Examination

Hypothesis  Usage of a Web-based educational tool will improve residents' scores on the American Board of Surgery In-Training Examination (ABSITE).

Design  Before and after trial.

Setting  Academic surgical residency in a tertiary care university hospital.

Participants  Nineteen postgraduate year 1 (PGY1) and PGY2 residents.

Intervention  All PGY1 and PGY2 residents were given unlimited access to a Web-based educational tool aimed at early-level residents in surgery. The educational tool covers the basic science of surgery and basic concepts of clinical surgery and uses a variety of educational techniques (readings, problem-based learning, case-based learning, and practice tests). Although residents were encouraged to complete the learning package, they were not required to do so.

Main Outcome Measures  The ABSITE scores and changes in ABSITE scores were compared with the amount of time residents spent using the program and the number of tutorials completed.

Results  There was no significant difference in ABSITE scores before or after the use of the Web-based educational tool. There was no significant relationship between use of the tool (either in total time or total tutorials) and ABSITE score. For PGY2 residents, there was a negative relationship between total time spent on the program and ABSITE score (linear regression analysis best-fit slope, −0.20 ± 0.1; r² = 0.33) and total tutorials completed and ABSITE score (linear regression analysis best-fit slope, −0.34 ± 0.22; r² = 0.25).

Conclusion  There was no demonstrable beneficial effect of a Web-based educational tool on cognitive knowledge of residents as measured by ABSITE scores.

Attainment of cognitive knowledge is a crucial part of residency education. Surgical residencies have used a combination of lectures, conferences, reading programs, and informal discussions to help residents obtain the cognitive knowledge base required to be independent surgeons. With the recent mandate to decrease resident work hours, there has been interest in using innovative methods to teach basic knowledge in surgery. Specifically, web-based modules have been advocated as a method to increase cognitive knowledge, which can be used while at work or on off-duty hours. These educational tools use a combination of reading material, video clips, simulated patients, and question-and-answer format to transmit knowledge. The Association of Program Directors in Surgery (Bethesda, Md) is currently embarking on a project to produce a web-based multimedia educational tool for residents. BeST Resident (Harvard Medical International, Cambridge, Mass) is a commercial Web-based education program aimed at residents in the early stages of postgraduate surgical education. Through a variety of methods (textbook readings, problem-based learning, case-based learning, and practice tests), participants are exposed to the basic scientific knowledge expected of a junior-level resident and covered by examinations such as the American Board of Surgery In-Training Examination (ABSITE) and the Fellowship of the Royal College of Surgeons examination.

Methods to assess an adequate knowledge base have included individual residency written and oral examinations, evaluation by preceptors on clinical rotations of a resident's knowledge base, and use of the ABSITE. The ABSITE has become widely accepted by surgical residencies as a primary method of evaluating residents' cognitive knowledge base and has been used by our residency since its inception in 1975 as part of the evaluation process in assessing residents' cognitive knowledge. This study is an attempt to see whether use of BeST Resident affected residents' scores on the ABSITE.

Beginning in June 2004, unlimited access to BeST Resident was provided to postgraduate year 1 (PGY1) and PGY2 residents. Residents were told that they were expected to complete all tutorials in the program during the 2004 through 2005 academic year. Resident usage of the program was monitored by the program director, and emails were sent to residents with low usage reminding them of the importance of studying for the ABSITE and the availability of this program, but no actual requirement of usage was made, and no adverse consequences of nonusage were outlined. On the day of the ABSITE, each resident's total hours of usage of BeST Resident and number of tutorials completed were tabulated. Nineteen PGY1 and PGY2 residents took the ABSITE in 2005, all of whom had unlimited access to BeST Resident. Nine of these residents were PGY2 and had taken the ABSITE in 2004, allowing evaluation of the change in score with BeST Resident.

The ABSITE scores were reported by Standard Score from the ABS; percent correct and percentile score were not used in any of the analyses. Statistical analyses were performed using GraphPad Prism (GraphPad Software, Inc, San Diego, Calif). Correlation of total hours of BeST Resident usage or tutorials completed with ABSITE score or change in score was by linear regression analysis.

Mean ABSITE scores for PGY1 residents rose from 471 to 481 with the institution of BeST Resident, but this change was not statistically significant. Likewise, although mean ABSITE scores for PGY2 residents fell (510 to 495), this difference was not significant. Of note, we were unable to demonstrate a change in ABSITE score related to decreased work hours (Table).

Nineteen residents had unlimited access to BeST Resident and used the program for a mean of 13.62 hours (median, 11.5 hours; range, 0-58.5 hours), completing a mean of 30.31 tutorials (median, 20; range, 0-100). Linear regression analysis of ABSITE score vs total time spent on BeST Resident revealed a best-fit slope of −0.03 ± 0 (r² = 0.03). Linear regression analysis of ABSITE score vs total tutorials completed on BeST Resident reveals a best-fit slope of 0.00 ± 0.09, (r² = 0.00). Thus, there was no significant relationship between BeST Resident usage (as measured by time or total tutorials) and ABSITE score.

Nine residents had unlimited access to BeST Resident and had taken the ABSITE in 2004 and 2005, thus allowing evaluation of change in standard score. Changes in these residents' scores ranged from −70 to +137, with a mean of +12.9 standard score points. These residents used BeST Resident a mean of 16.22 hours (median, 10.5 hours; range, 0-58.5 hours) and completed a mean of 35.9 tutorials (median, 15; range, 0-100). Linear regression analysis of change in score vs total time spent on the program revealed a best-fit slope of −0.20 ± 0.1 (r² = 0.33) (Figure 1). Linear regression analysis of change in score vs total tutorials completed revealed a best-fit slope of −0.34 ± 0.22 (r² = 0.25) (Figure 2). Thus, there was a negative relationship between BeST Resident usage and change in ABSITE score.

The process of learning surgery is long and arduous. It involves the acquisition of cognitive knowledge, judgmental skills, technical skills, and interpersonal skills. With the mandate to decrease the number of hours spent in the hospital by residents, we have become interested in techniques to shift some of this learning to nonwork, out-of-hospital hours. Specifically, cognitive knowledge has historically been an area of surgical learning relegated largely to off-hours time and left to the discretion of the individual resident. Cognitive knowledge thus seems to be a good area to try newer teaching techniques such as a Web-based teaching tool. Such a tool provides a method to access various teaching techniques (readings, video clips, simulated patients, practice tests) outside the confines of the normal work/educational environment and hours yet maintain the ability for faculty to monitor progress in usage.

Measuring the acquisition of this knowledge is difficult. Although the ABSITE is certainly an imperfect measure of cognitive knowledge, it is widely used as 1 measure of this acquisition, and it is standardized and reproducible.¹ Thus, we chose to use it as our measure of whether a web-based education module would help improve our residents' knowledge base and improve their scores on this examination. We could find no relationship between BeST Resident usage and ABSITE score but did demonstrate a weak negative relationship between such usage and the change in residents' score between PGY1 and PGY2. Of concern is the low level of usage of the program by our residents. Residents stated that they felt that the web-based program was not as comprehensive nor as detailed as textbook review, and thus the majority of residents elected to use textbook review rather than BeST Resident as their primary method of study. Perhaps a different web-based tool would be of more value to our residents, but based on our experience, we cannot presently recommend such educational methods because it is not completely clear whether it is the content or the format of the program that residents found unappealing.

Scores on the ABSITE are commonly used as measures of cognitive knowledge of surgery, and they have been used as a means to evaluate various teaching methods. Previous studies have demonstrated a correlation of ABSITE scores to residents' study habits² and to programmed reading.^3,4 Although some studies have shown a relationship with conference attendance,^5,6 others have not.⁷ Resident-performed preparation and presentation of educational lectures has been shown to improve scores on the Thoracic Surgery in Training Examination,⁸ reinforcing the concept that it is the individual resident's effort in self-education that most affects performance on the ABSITE. Previous authors have shown improvements in ABSITE score with work-hour limitations⁹: this was not the case for our residents, whose scores actually decreased following the implementation of work-hour limitations. Finally, we have previously found that residents with a history of poor performance on the ABSITE improved their scores by 50 standard points (half of a standard deviation) following attendance at the Association of Program Directors in Surgery basic science course (C.M.F., unpublished data, 2000). Unfortunately, expense makes this a very limited option.

Several limitations of our study need to be recognized. First, this is a rather small sample to judge the validity of a specific learning tool. A larger, cooperative study would be more reliable in showing the effectiveness of a web-based teaching tool. Second, the actual usage of BeST Resident by our residents was modest. Residents were not threatened with negative action for nonusage and thus were not truly “required” to complete the program or even to try it. Although this may seem an unconventional policy, it is in keeping with the educational philosophy of surgical residency: we provide the tools for education, but the resident must use them. Most of our residents felt that this was not a valuable tool for them, hence the low usage.

It is possible that had residents been required to complete the entire educational package of BeST Resident, then scores may have increased, but this is pure speculation. In fact, the individual in our residency who had the greatest total time (58.5 hours) and tutorials completed (100) had the greatest decrease in ABSITE score (−70 standard score points). In summary, we could not demonstrate a beneficial effect of a Web-based educational tool on cognitive knowledge of our residents as measured by ABSITE scores.

Correspondence: Charles M. Ferguson, MD, Massachusetts General Hospital, 15 Parkman St, Boston, MA 02114 (cferguson@partners.org).

Accepted for Publication: October 4, 2005.