Malloy MH, Speer A. A Comparison of Performance Between Third-Year Students Completing a Pediatric Ambulatory Rotation on Campus vs in the Community. Arch Pediatr Adolesc Med. 1998;152(4):397-401. doi:10.1001/archpedi.152.4.397
To compare the performance of third-year medical students who completed the ambulatory component of their pediatric rotation in a community setting with the performance of third-year medical students who had their ambulatory experience on campus.
As part of a pilot project to implement a third-year Multidisciplinary Ambulatory Clerkship, 61 third-year medical students spent 12 weeks rotating through the primary care disciplines of family medicine, internal medicine, and pediatric practitioners' offices at sites distant from the university campus while 127 students remained on campus for their ambulatory experiences in these disciplines. The components of the overall pediatric grade consisted of a clinical performance evaluation in the ambulatory setting (4 weeks), a clinical performance evaluation on a 4-week inpatient rotation, and a grade from a multiple-choice final examination.
The overall mean±SD final pediatric grade of students receiving their ambulatory pediatrics experience in the Multidisciplinary Ambulatory Clerkship was 86.5±3.4 compared with 88.0±3.4 for students receiving their ambulatory experience on campus (P<.007). This difference was accounted for by performance on the written final examination. Multidisciplinary Ambulatory Clerkship students had a mean±SD score of 78.9±8.3 and a failure rate of 18% compared with a mean score of 83.7±8.1 and failure rate of 3.9% for students who remained on campus for their ambulatory experience (P<.001 for both comparisons). No differences were noted between the 2 groups on their clinical performance evaluations for their ambulatory or inpatient experiences.
These data suggest a difference in the learning experience between students receiving their pediatric ambulatory experience in the community vs on campus. Differences in exposure to structured learning experiences that occurred more frequently on campus might account for some of the difference in final examination results. Development of a standardized, structured learning experience across community sites would seem to be an appropriate means of enhancing learning in the community setting.
THE TRANSITION of medical education from the inpatient setting to the ambulatory setting is being pushed forward by several forces. Recognition that the actual practice of medicine occurs mostly in an outpatient setting, the effect of managed care on reducing the number of patients available for teaching within the inpatient services, and the burden of increased direct patient care imposed on medical school faculty have all contributed to the movement of medical education out of the university hospital.1- 5 Reviews of the learning experiences of students in the ambulatory setting, as well as guidelines for medical education in community-based practices, have been published.6- 8 However, whether this transition of medical education to the community will provide students with knowledge and skills comparable to the knowledge and skills acquired by students trained in university settings is just beginning to be explored.9- 11
During the pilot year of a Multidisciplinary Ambulatory Clerkship (MAC), we had the opportunity to compare the performance of third-year medical students electing to serve the ambulatory component of their pediatric rotation in the community with that of third-year students remaining at the university. In this article we compare the performance of these 2 groups of students and attempt to understand the basis for the observed differences.
In June 1995, the University of Texas Medical Branch, Galveston, began piloting a MAC for third-year medical students. The MAC was designed as a 12-week rotation that provided approximately 4 weeks in each of the following primary care disciplines: family medicine, internal medicine, and pediatrics. The rotation was scheduled so that students would have their ambulatory experiences in each of these disciplines integrated across 12 weeks instead of spending 4-week blocks in each discipline. As a result, MAC students could spend up to 1 week at a time in any one discipline before moving to another discipline, or they could spend 11/2 days a week in each discipline, or any number of other permutations that would provide an equal distribution of time among the disciplines during the 12-week period.
The MAC sites were private physician offices in communities located 50 miles (80 km) or more away from the university campus. The sites were spread as far north as Nacogdoches, Tex (≈220 miles [352 km] from Galveston), and as far south as Victoria, Tex (≈250 miles [400 km] from Galveston). This required that the students reside at the community site for the 12-week period.
During the pilot year of the MAC, 61 students volunteered for a community ambulatory experience in the 3 primary care disciplines. One hundred twenty-seven students remained on campus and received their ambulatory experience within each of the 3 primary care disciplines in 4-week block rotations. Pediatrics was an 8-week rotation with 4 weeks spent on an inpatient rotation and 4 weeks spent on an ambulatory experience in the university hospital clinics. The university hospital clinic experience included subspecialty clinics. The MAC students had their 4 weeks of pediatric ambulatory experience at a community site.
During the pilot year, each discipline separately evaluated the students. The pediatric grade for a MAC student consisted of the community pediatric preceptor evaluation for ambulatory clinical performance, an inpatient attending's evaluation of the student's inpatient clinical performance, and the grade obtained on a departmental multiple-choice final examination. For a student remaining on campus the pediatric grade consisted of an assigned mentor's evaluation of the student's ambulatory clinical performance, an inpatient attending's evaluation of the student's inpatient clinical performance, and the grade obtained on the final examination. The written final examination for both groups consisted of 50 to 55 questions and was derived from a pool of 200 questions written by the pediatric faculty. Most of the questions were submitted to the pool prior to the start of the academic year under study. The final examination was administered at the conclusion of the entire pediatric experience. Thus, for MAC students, the final examination would come at the conclusion of a 16-week period, 12 weeks of the MAC and 4 weeks of inpatient experience, while for the students remaining on campus the examination would come at the conclusion of an 8-week experience (Table 1). The MAC students could have had their 4-week inpatient experience before or after their 12-week experience in the community.
Differences were noted in the experience of the students who remained on campus compared with that of students receiving their ambulatory pediatric experience in the MAC. At the clinical level, students remaining on campus had the opportunity to rotate through subspecialty clinics. There were many structured learning opportunities on campus that included a weekly problem-solving case conference designed specifically for the students, a weekly radiology conference, grand rounds, and twice-weekly resident conferences that the students could attend. No structured conferences were organized by the MAC for the students in the community. None of the final examination questions came directly from any of the on-campus conferences.
Clinical evaluations of student performance for both the ambulatory and the inpatient experiences used the same evaluation instrument that inquired about the students' skills in taking medical histories and performing physical examinations, written and oral presentation skills, knowledge base, interpersonal skills, and professional skills. Grades for the students were recorded on a computerized spreadsheet. This information was exported into an SAS file (SAS Institute Inc, Cary, NC) and underwent statistical analysis using the SAS System.12 The US Medical Licensure Examination (USMLE)– Part I and USMLE-Part II scores were obtained from the registrar's office and merged into the SAS data set. All analyses were conducted in a fashion that assured student anonymity. χ2 Tests for general association were used for discrete variables and Student t tests for continuous variables. P ≤.05 was considered to be statistically significant. The values are expressed as the mean ± SD.
We examined several student characteristics by type of ambulatory rotation to determine if any specific type of student was selecting the MAC rotation. No differences were noted between the students rotating on campus vs in the community for sex, age, residency site selection, or type of residence selected (Table 2). The mean for the overall scores and the individual components of the grade for the 2 groups are given in Table 3. The mean±SD overall score for the MAC students was 86.5±3.4, significantly less than that for the on-campus students, 88.0±3.4 (P<.007). Of the individual components of the grades, only the final examination score was significantly different between the 2 groups. The mean score for the MAC students was 78.9±8.2 compared with 83.7±8.1 for the on-campus students (P<.001).
The examination failure point was set at a score of less than 70. This failure point was approximately 2 SDs from the mean final examination score of the previous year, which was 84±8. That final examination was derived from essentially the same pool of questions. The examination failure rate was significantly increased among the MAC students (18.0%) compared with the on-campus students (3.9%) (P<.001). Thus, performance on the final examination accounted for the lower overall mean score for the MAC students.
In all but 1 instance the MAC students received a different examination from the students who remained on campus. In that 1 instance, however, the on-campus group and the MAC group were both finishing their respective 8-week and 16-week experiences. For this particular examination, the mean final examination scores for the on-campus students was 80.3±7.7 compared with 81.2±10.1 for the MAC students (P=.75). However, only 1 (3.2%) of the 30 on-campus students failed the examination compared with 4 (25%) of 16 MAC students.
We were concerned that one potential confounder in the students' performance might be the order in which they proceeded through the various components of the pediatric rotation. For example, some of the MAC students might have had poorer test performance because they had their 4-week pediatric inpatient experience first and did not take their final examination until an additional 12 weeks of the MAC rotation had been completed. We thus compared the failure rate between MAC and on-campus students by the order of their rotation just prior to taking their final examination. For students who had their inpatient rotation just prior to the final examination, the failure rate was 0 (0%) of 35 for the on-campus students compared with 4 (13.3%) of 30 for MAC students (P=.03). For students who had their ambulatory rotation just prior to the examination, the failure rate for the on-campus students was 5 (5.4%) of 92 compared with 7 (22.6% of 31) for MAC students (P=.006).
We were also concerned that performance on the final examination might have been influenced by the quarter of the year in which the student was enrolled in pediatrics. In the first quarter of the year the examination failure rate for the on-campus students was 11.8% compared with 13.3% for the MAC students (Table 4). However, for the remaining quarters of the year, the examination failure rate for the on-campus students was almost nil while the MAC student examination failure rate ranged from 13% to 25%.
To adjust for any potential confounding of the relationship between the type of rotation and the final examination score, we used an analysis of variance model in which we included the type of rotation (on-campus vs MAC), sex of the student, residency site selection, type of residency selected, student age, order of rotation (ambulatory or inpatient last), and quarter of the year in which the rotation was taken. Only type of rotation (on-campus vs MAC) was significantly associated with the final examination score (P<.001).
Because students going out on the MAC were a volunteer population, we were concerned that a population of students might have volunteered who, for whatever reasons, were not good test takers. We attempted to assess test-taking ability before the third-year pediatric rotation and after the rotation by examining the USMLE-Part I and USMLE-Part II scores. The mean USMLE-Part I score for the on-campus students was 205±19 compared with 202±20 for the MAC students (P=.45). The mean USMLE-Part II score for the on-campus students was 210±22 compared with 204±20 for the MAC students (P=.11).
These data suggest a difference in performance on the final examination between students who had their pediatric ambulatory experience in the community compared with those who remained on campus. The reasons for this difference are likely to have been multiple and not attributable to one single issue, thus they should not be construed as evidence that the clinical experience in the community was necessarily inferior to the on-campus experience. There was, however, something different either about the experience that the students had in the community or about the population of students electing to go out into the community that separated their performance on the final examination from students remaining on campus. Regardless of the timing of the ambulatory experience relative to the final examination, MAC students failed at a greater rate than students remaining on campus. In addition, except for the first quarter of the year, MAC students failed at a greater rate for the remaining 3 quarters. The failure to observe a difference in performance in pediatrics relative to the time of year has been reported by Hampton et al.13
Another sources of differences between the experiences of the 2 groups was the availability of structured learning experiences for the on-campus students. We have no quantitative measure of the degree of participation in the on-campus activities, however, so we cannot determine any dose-response effect. As others3,6,7 have suggested as a means of enhancing community-based ambulatory experiences, we are exploring ways to provide structured learning opportunities to students in the community to standardize learning experiences across community sites. Several studies6,9,10 have addressed the issue of the equivalence of student experience and performance across different settings. Friedman et al9 compared an obstetrics and gynecology rotation at a university hospital to the same rotation at 3 community hospitals. The authors observed differences in learning activities at the various sites, but did not document any differences in performance at the conclusion of the rotation. Feltovic et al6 surveyed the internal medicine chairpersons of 129 medical schools in 1986 and reported that only 27% of the programs offering an ambulatory rotation included any structured educational experiences to complement clinical patient-care experiences. Kurlandsky et al,10 evaluating student performance on a pediatric rotation offered in 5 different community settings, reported a significant difference in student performance on the final examination in 1 of the 5 communities. The authors could not, however, attribute the difference to any specific discrepancy in student experience observed in that single community.
None of the studies described earlier, however, address the critical issue that we are describing in this article. That is, what will be the impact on student knowledge and skill acquisition once ambulatory experiences are moved from the university to a community in which predominantly private practitioners will be used as preceptors? Although in this article we infer that a difference in performance on the final examination may reflect knowledge acquisition, we cannot absolutely prove that this reflects differences in overall competency. The possibility that a group of students with poorer test-taking skills selected the community rotation is a legitimate hypothesis that is weakly supported by observing the trend in differences on USMLE scores. The possibility that the difference in performance was related to the lack of availability of a more standardized and structured experience across sites, however, suggests the need to pursue more aggressively the design, implementation, and evaluation of such experiences.
Several forces have come to bear on the movement of medical education into the community. The Association of American Medical Colleges' report, "Physicians for the Twenty-First Century" noted that "although fewer than five percent of all physician-patient contacts result in hospitalization, clinical clerkships are predominantly based on hospital inpatient services."1 This report, however, does not make specific recommendations to move clerkships into the community, but rather lists the disadvantages associated with the predominant custom of confining the education of medical students to the teaching hospital. The report concludes that "developing and maintaining hospital inpatient and outpatient and community settings appropriate for required medical student clerkships in the major clinical disciplines will require both ingenuity and the expenditure of resources."1 The report offers little guidance on just what that appropriate mix of inpatient, outpatient, and community medical education might be. Nevertheless, this report is cited in several articles as a sentinel event in raising general awareness that moving medical education into the community is a laudable goal.7,8
Other forces favoring the movement of medical education from the teaching hospital to the community include the shift of patient care from the inpatient to the outpatient setting that is being encouraged by managed care plans. The resulting decrease of inpatients available for medical education and the threat of downsizing the academic center workforce have also contributed to the attractiveness of teaching in community settings.4,5
Another factor in the movement of medical education to the community setting is the premise, attended by mixed research results,14- 16 that education in the community will influence student practice choices to favor primary and community care. Kurlandsky et al,10 in reviewing the performance of pediatric clerkship students in 5 different clinical settings, failed to document any greater likelihood that students who spent 50% of their time in private office settings for their clinical experience would select a pediatric residency. Kassebaum et al,16 in a review of the Association of American Medical Colleges databases for 1995 medical school graduates, observed that plans to pursue certification in family practice or an unspecified generalist career could be influenced by experiences in family medicine or an ambulatory primary care rotation, while career decisions to go into general internal medicine or general pediatrics were not. Kaplowitz et al15 surveyed students at the beginning and end of their pediatric clerkships in 11 medical schools from July 1, 1992, through June 30, 1993, and observed that the clerkship experience positively influenced interest in a pediatric career. Inpatient experiences, however, were more highly correlated with changes in career decisions than were outpatient experiences. In particular, private practice preceptors received lower mean ratings.
Thus, the outcome of arguments for pushing medical education into the community appears somewhat mixed. Movement into the community for economic reasons and because "that is where the patients are" may be practical, but will not ensure an optimal education. Continued monitoring of the effect of this change in the medical education process is necessary and important for curriculum development. Our limited experience should serve as a reminder of the need for further evaluation.
Accepted for publication October 21, 1997.
This study was funded in part by a Robert Wood Johnson Foundation Generalist Physician Initiative Award, Princeton, NJ.
Presented in part at the Annual Meeting of the Council on Medical Student Education in Pediatrics, Phoenix, Ariz, March 22, 1997.
We thank Richard Lewis in the Registrar's Office of the University of Texas Medical Branch, Galveston, for his help in obtaining student information.
Editor's Note: It's nice to have documentation that it's not sufficient merely to place students "where the action is" for them to learn. I'm sure medical educators know this, but we continue to ignore it.—Catherine D. DeAngelis, MD
Corresponding author: Michael H. Malloy, MD, MS, Department of Pediatrics, University of Texas Medical Branch, 301 University Blvd, Galveston, TX 77555-0526.