Civetta JM, Morejón OV, Kirton OC, Reilly PJ, Serebriakov II, Dobkin ED, D'Angelica M, Antonetti M. Beyond RequirementsResidency Management Through the Internet. Arch Surg. 2001;136(4):412-417. doi:10.1001/archsurg.136.4.412
An Internet application could collect information to satisfy documentation required by the Residency Review Committee. Beyond replacing a difficult and inefficient paper system, it would collect, process, and distribute information to administration, faculty, and residents.
An integrated residency of 18 services at a university teaching hospital with 4 affiliated institutions.
Residency administrators, faculty, and residents.
The application included a procedure recorder, resident evaluation of faculty and rotations, goals and objectives (stratified by service and resident level), and matching faculty evaluation of residents with these goals as competencies. Policies, schedules, research opportunities, clinical site information, and curriculum support were created.
Main Outcome Measures
Degree of compliance with Residency Review Committee standards, number of deficiencies corrected, and quantity and quality of information available to administration, faculty, and residents.
The Internet system increased resident compliance for faculty and rotation evaluations from 20% and 34%, respectively, to 100%, which was maintained for 22 months. These evaluations can be displayed individually, in summary grids, and as postgraduate year–specific averages. Faculty evaluations of residents can be reviewed throughout the system. The defined category report for procedures, which had deficiencies in the preceding 6 years, had none for the last 2 years. The Internet application provides Accreditation Council for Graduate Medical Education–validated operative logs to regulatory agencies.
A Web-based system can satisfy requirements and provide processed data that are of better quality and more complete than our paper system. We are now able to use scarce time and personnel to nurture developing surgical residents instead of shuffling paper.
IT WAS A HUMBLING and even a frightening experience for the fully accredited general surgery residency at the University of Connecticut, Farmington, to be placed on probation after a site visit by the Residency Review Committee for Surgery (RRC). Academic health centers already faced with severe financial problems must contend with increased expenditures for midlevel practitioners to ensure adequate patient care so that residents can fulfill their educational goals. As teaching faculty, both full-time and volunteer, also have had to increase their clinical workload, time for didactic sessions and mentoring of residents can easily become a casualty.
The increased emphasis in the requirements for general surgery1 on the educational content of residency programs reflects awareness that these pressures could have adverse effects. Perhaps the best understanding of the intent of requirements can be gained from reading the instructions for completing the program information form for residency training and programs in surgery.2 While there are explicit requirements for collecting information and for documentation to improve the educational quality of the program, this information must be used to affect change and thus must be processed into usable formats and be available throughout the residency.
As we attempted to correct cited deficiencies, the limitations of the traditional paper system became apparent, especially if we wished to go beyond minimum requirements and develop concerted efforts to improve the educational value of our residency. We chose, therefore, to discard the time-consuming and inefficient paper system that relied on the administration to process and distribute the data collected. We created instead a new paradigm that used an Internet application to collect, process, and distribute the information to the administration, faculty, and residents, as well as provide processed output suitable for regulatory bodies. Given that there were 5 institutions (the university hospital and 4 affiliated institutions), 18 services, and 44 residents in the integrated general surgery residency, this paradigm shift was envisioned as a measure that would simultaneously meet current requirements and enable utilization of data in ways that we considered desirable but were beyond the capabilities of our paper system. However, we were not attempting to create a technological solution per se; rather, we heeded the words of Albert Einstein3: "We should be on our guard not to overestimate science and scientific methods when it is a question of human problems." The process of effecting change started with the commitment of the program administration, the support of the Resident Education Committee and hospital administrations, and the participation of both the faculty and residents at all levels to design and implement the Internet application.
The deficiencies in the residency program included inadequate documentation or failure to provide adequate documentation, and lack of commitment to create, monitor, and maintain the educational program. The Internet application was seen as a vehicle for helping to correct these deficiencies. To create the application, the university and the affiliated hospitals agreed to support the individuals with the necessary skills and dedication. These individuals included physician-educators, computer-literate physicians in the residency administration, a system administrator, and various individuals and companies contracted to perform specific tasks (Table 1).
The next step involved extracting and organizing the elements in the program information form instructions that could be created on the Internet. E-mail listserves of faculty and residents were created so that all would be informed when new information was available. A dedicated server was purchased to support the Web site. Informational elements were constructed by the residency coordinator, the associate program director for development, and the system administrator. These items were saved from FrontPage (Microsoft Corp, Redmond, Wash) to the Intranet. Databases were constructed to be dynamic; after each new evaluation was entered, for example, the queries that automatically generated reports incorporated the new data entered. Thus, all the material in the databases was reflected in reports of evaluations of residents, although faculty and rotation evaluations were posted only in blocks to preserve resident anonymity.
A task force of faculty and residents created overall goals for the program, general characteristics that would be assessed on all rotations, and specific cognitive and technical goals for each service, and divided them into junior, midlevel, and senior resident levels. These goals and objectives were abstracted from the Association of Program Directors' Surgical Resident Curriculum, second edition,4 and posted in the Internet application. This task force created evaluations of residents by the faculty based on the goals and objectives, including the general characteristics and the service and resident level–specific technical and cognitive elements.
A form to evaluate the integrated basic science clinical curriculum was added as well. All evaluation forms were created in Microsoft Access (Microsoft Corp). The forms were designed with pull-down menus for items that appeared in each evaluation. Comments could be entered as free text. Reports were generated from the databases as re-creations of the individual evaluations plus summaries and averages; the comments could be recalled by any user. A hierarchy consisting of 7 levels was created by the system administrator. Information was then available only on a "need to know" basis. An Internet operative procedure recorder based on the personal computer program supplied by the Accreditation Council for Graduate Medical Education was created. Data entry forms provided individual surgeon and institution information as well as exporting capabilities to create files, defined categories, and statistical analysis for both residents and the program.
Policies concerning documentation requirements were created by another resident and faculty task force. The policies were approved by the Resident Education Committee, consisting of 20 faculty members (all 5 hospitals were represented) and 7 elected residents. All policies relating to the residency, RRC requirements, and the Capitol Area Health Consortium, Farmington, which employs the residents, were made available online, and notification of any changes was distributed through the listserves.
The various elements were phased in sequentially. Recognizing that changing behavior and, especially, adding consequences cannot be accomplished overnight, the administration allowed the residents 3 to 4 months to become familiar with the operation of the Web site. The residency coordinator, associate program director for development, and system administrator made themselves available by phone, by e-mail, in orientation sessions, and individually to help with the transition. The same resources were used to help the faculty members adjust to the transition to the electronic medium. A "countdown" to actual conferences began about 2 months before the consequences of noncompliance were actually used. Messages, both electronic and verbal, were repeated weekly to emphasize the need for resident compliance.
The process of transforming a paper system to the Internet application began in 1998, accelerated through 1999, and continued in 2000 (Table 2). Funding was provided by the Ludwig J. Pyrtek Fund at the Hartford Hospital, Hartford, Conn; the University of Connecticut Health Center; and faculty salary support at the individual hospitals. Table 3 shows an estimate of the costs.
Consolidating and posting information on the Internet application had a major effect on true integration of the residency. The long countdown period and repeated messages resulted in about 90% compliance immediately. Almost every one of the last 10% responded to a final verbal notification, so that only a few residents actually had the outlined first consequence—loss of 1 vacation day for each day that they were delinquent (after a 2-week grace period and 1 final week after verbal notification of delinquency).
The faculty members, especially volunteer faculty, had somewhat more difficulty in complying with the deadlines for entering evaluations. Two factors were identified: limited access to the Internet and a moderate "computer phobia." Both issues were addressed at each of the hospitals, with marked improvement. The competency-based evaluations by faculty were considered a significant improvement over the single evaluation form previously used. Beyond the compliance with documentation, new attributes were added. If a resident showed signs of excessive stress, an e-mail was automatically sent to the program director, associate program director, and residency coordinator. If the resident was "clearly deficient" in a category, checking an entrance interview box generated an e-mail again to the program director, associate program director, and residency coordinator, and to the service chief of the resident's next rotation. This e-mail contained the comments box from the evaluation.
The low rate of return of faculty and rotation evaluation by residents was one of the deficiencies cited by the RRC during their site visit. Before the construction of the Internet application, increased administrative attention (announcements at all conferences and paging of residents who were delinquent) and forms that could be sent and retrieved electronically by e-mail were attempted. Although improvement was noted,5 the Internet application resulted in 100% compliance, a rate that has been maintained for 22 months (Table 4). The use of a numeric code for entry created accountability while preserving anonymity; the residency coordinator and system administrator have the only lists matching the codes with residents' names. When one rotation had to be eliminated because of an inadequate number of residents, rotation evaluation data allowed the Resident Education Committee to eliminate the rotation that had been rated to have the least educational value (Table 5). The Web-based procedure recorder enabled each case to be entered in approximately 30 seconds. An administrative policy was added to the Residents' Manual requiring concurrent entry. Deficiencies in the defined category report had been present for 6 years. When the new procedure recorder was developed, the chief residents for 1998-1999 completed all the defined categories by May 23, 1999, and by February 10, 2000.
The basic science curriculum is also an RRC mandate. The program information form asks for a listing of topics and speakers. The faculty-resident task force designed a comprehensive curriculum, based on Essentials of Surgery: Scientific Principles and Practice.6 The Internet application provides a calendar, learning objectives, PowerPoint presentations, and journal articles in downloadable formats. Finally, conference evaluation forms are required for residents to document attendance and have been used by the task force to make improvements in the integrated curriculum. The many different methods of presenting information have changed the resident's role from passive, ie, listening to didactic presentations, to an active adult learning mode. The program evaluation was not cited as a deficiency or created as an initial Internet application. However, sequential 6-month program evaluations by residents indicated improved satisfaction with the new evaluation system and basic science curriculum, especially related to feedback that was only possible through the Internet application.7
Eighteen months after being placed on probation, the program was reevaluated. Probation was removed and the next visit will be scheduled in 4 years. The Intranet application has helped us advance "beyond requirements" in other ways: training the surgeons of the future to use computer-based communication, documentation, reporting, logging, analyzing, and presenting of cases and lectures at conferences. Performing adequate and timely documentation will likely be a habit that they may only really appreciate after residency when, as surgeons, they seek credentials and hospital privileges and become providers for insurers.
For those interested in the application, the address is http://www.surgweb.uchc.edu. Visitors can enter the Intranet portion (a button on the home page that takes the user to a second page to either register or enter the Intranet) by using "facultyguest" as both user name and password. They can see all the information portions, but entry and review of all evaluations and other data are protected by a hierarchy of privileges contained within the application.
Our integrated general surgery residency faced the challenge of correcting deficiencies that resulted in probation by creating an Internet application. Data collection and processing allowed us to correct problems with evaluations. The ability to provide processed information throughout the integrated residency fostered growth of the residents by reinforcing good performance and being able to quickly respond to deficiencies. Each new element led to ideas that go beyond merely meeting requirements. These ideas attempt to create an online continuous quality improvement process and to meet the challenges of surgical education in the future.
The University of Connecticut School of Medicine, Farmington; New Britain General Hospital, New Britain, Conn; Saint Francis Hospital and Medical Center, Hartford; Connecticut Children's Medical Center, Hartford; the Hartford Hospital, Hartford; and the Ludwig J. Pyrtek Fund of Hartford Hospital provided support for the development of the Internet application.
Presented at the 81st Annual Meeting of the New England Surgical Society, Boston, Mass, October 8, 2000.
Corresponding author: Joseph M. Civetta, MD, Department of Surgery, University of Connecticut School of Medicine (MC3955), 263 Farmington Ave, Farmington, CT 06030-3955 (e-mail: email@example.com).
Michael Stone, MD, Boston, Mass: In this paper, Dr Civetta and his colleagues have described their use of their Internet Web site to enhance data collection, communication, and documentation for Residency Review Committee requirement compliance in a large, complex residency program with, as he has described, 44 residents scattered over 18 services in 5 hospitals. In such a system, the need to facilitate communication and data collection is obvious. In using the Internet-based data collection system, resident compliance in completing faculty and rotation evaluations, 2 of the many RRC requirements, was 100%. Every program director would tell you that getting 100% of their residents to comply with any requirement is a remarkable feat.
Educators have for a long time envisioned the use of computers for medical education. In surgical education, in particular, the potential for the use of graphics or 3-dimensional modeling to teach anatomy and surgical technique is enormous but remains an unfulfilled goal at present. What we have learned from that process, however, is that simply computerizing paper-based tasks is not enough. The financial and time costs of system development have made it clear that we need to look past the panache of computers and find the value added by the electronic media. The value added by the Internet is rapid communication, meaning the posting and reading of text, data, and graphic content. Dr Civetta has described the ability to update important documents, such as his residency policy manual and the list of educational objectives, with the implications that these are read by their residents. My first question is, do you track the number of hits on the elements of your Web site, particularly those that document important information that residents must know?
By far the most common use of the Internet, as we have learned from the business world, is for e-mail. E-mail clearly can be a tremendous time saver, but I have been impressed that people will write things in e-mails that they would never say face-to-face. In other words, how we communicate with one another alters what we are willing to communicate. My next question relates to the rapid communication to residents of their evaluation by the faculty. Research on feedback indicates that medical faculty are reluctant to give negative feedback, especially verbally and face-to-face. Your residents are informed almost immediately of electronic reported deficiencies and called in for discussion with you. Do you think this alters your faculty's willingness to be frank in their evaluations, and how?
You also provide that evaluative data to the resident's next rotation director. This is a good idea, but there are some educators who worry that this might prejudice the resident's evaluation on the new rotation. What are your thoughts about immediate evaluation as opposed to looking at performance over a longer period of time?
As you have seen, Dr Civetta has used a carrot-and-stick approach in getting his hardworking residents to fulfill the mundane task of filling in forms. He has been able to provide a computer for each resident and has modified the cumbersome RRC-required ACGME [Accreditation Council for Graduate Medical Education] surgical operative log software to facilitate data entry. These are the carrots. The stick is that the residents lose a vacation day for each administrative day they spend completing delinquencies. The combination of these efforts has resulted in 100% compliance. We have also achieved 100% compliance with our residents, but I now find it more challenging to get our hardworking faculty to provide us with timely data. What is your experience in this regard? And what is your stick for the faculty?
What did it cost to develop and maintain this system? And have you been able to demonstrate cost savings or free up program administrators to perform other tasks?
Finally, 2 comments. If you are a program director or chairman in this electronic age and you do not have a Web site, you had better build one. As you can see, you are way behind the times. This is a wonderful Web site at the University of Connecticut.
Second, more has gone into enhancing the quality of surgical education at the University of Connecticut than Web site development. Computers do not solve educational problems; people do. In his presentation this morning, Dr Civetta has pointed out that quality surgical education results from the commitment of the chairman along with the support of the hospital administration and the participation of the faculty and the residents.
Dr Civetta: We do not track hits per se, but we do have a policy that has a graduated number of steps before we actually pull vacation, and the first step is that all residents are sent an e-mail a week before the end of the rotation saying it is that time of month again, get all your documentation up-to-date, and then 2 weeks later the residency coordinator or the secretary in the office actually checks to make sure that the information has been submitted. If not, the residents are given an e-mail and also paged to let them know that they have a week to catch up. It is only after that time that they are considered deficient. What has been striking is that, over the period of the first year, now about 95% of the residents need no urging and those who get the page that they are within 1 week of becoming deficient correct it that afternoon. We have not had anybody lose a vacation day in about a year.
The idea of the information being available in an impersonal way was addressed by the task force that addressed trying to redo the evaluation system. The residents are given an exit interview on every rotation at this point and it is after that exit interview that the form is filled out, so that we do believe that this face-to-face interaction between the residents and the faculty member is the most important part of the evaluation process. It just helps to be able to review it and to be able to see a grid to see whether things are getting better or getting worse.
In terms of prejudicing someone, in fact that was one of the reasons why we started doing this, because we found that in the absence of concrete data and no one being able to actually see the evaluations, hearsay was being used—this resident is coming over to you and he is really doing a poor job—and that was prejudicing more people than having to write it down and people being able to actually see. So the evaluations seem to have actually improved in quality, although your very last question is the one that really is of concern, and that is, I have sticks with regard to the residents. I do not have sticks with regard to a volunteer faculty. There has, however, been increasing improvement in the compliance of faculty with filling out the evaluations as they get used to it. It is a learning curve, and the faculty probably is less used to it and less computer literate than most of the residents, so it has been easier to do it with the residents.
As far as the dollars, we predicted that question would be asked and it just occurred to me about the old J. P. Morgan question about owning a yacht: if you have to ask, it's too expensive. What we hope to do, since we spent a lot of time and effort, mostly people time, in developing this, is we are working currently on creating a generic Web site application, not only for surgical residencies, but the evaluation process could be applied to industry, to others, and we are going to try and contact a venture capital company to make it available. If so, the costs of being able to utilize it for an individual residency would be quite small.