Rattner SL, Louis DZ, Rabinowitz C, Gottlieb JE, Nasca TJ, Markham FW, Gottlieb RP, Caruso JW, Lane JL, Veloski J, Hojat M, Gonnella JS. Documenting and Comparing Medical Students' Clinical Experiences. JAMA. 2001;286(9):1035-1040. doi:10.1001/jama.286.9.1035
Author Affiliations: Center for Research in Medical Education and Health Care (Drs Gonnella and Hojat, Messrs Louis and Veloski, and Ms Rabinowitz), Office of the Dean (Drs Rattner and Nasca), Departments of Medicine (Drs J. Gottlieb and Caruso), Family Medicine (Dr Markham), and Pediatrics (Drs R. Gottlieb and Lane), Jefferson Medical College, Thomas Jefferson University, Philadelphia, Pa.
Context The decentralization of clinical teaching networks over the past decade
calls for a systematic way to record the case-mix of patients, the severity
of diseases, and the diagnostic procedures that medical students encounter
in clinical clerkships.
Objective To demonstrate a system that documents medical students' clinical experiences
Design and Settings Evaluation of a method for recording student-patient clinical encounters
using a pocket-sized computer-read patient encounter card at a US university
hospital and its 16 teaching affiliates during academic years 1997-1998 through
Participants A total of 647 third-year medical students who completed patient encounter
cards in 3 clerkships: family medicine, pediatrics, and internal medicine.
Main Outcome Measures Number of patient encounters, principal and secondary diagnoses, severity
of diseases, and diagnostic procedures as recorded on patient encounter cards;
concordance of patient encounter card data with medical records.
Results Students completed 86 011 patient encounter cards: 48 367
cards by 582 students in family medicine, 22 604 cards by 469 students
in pediatrics, and 15 040 cards by 531 students in internal medicine.
Significant differences were found in students' case-mix of patients, the
level of disease severity, and the number of diagnostic procedures performed
across the 3 clerkships. Stability of the findings within each clerkship across
3 academic years and the 77% concordance of students' reports of principal
diagnosis with faculty's confirmation of diagnosis support the reliability
and validity of the findings.
Conclusions An instrument that facilitates students' documentation of clinical experiences
can provide data on important differences among students' clerkship experiences.
Data from this instrument can be used to assess the nature of students' clinical
Medical students are exposed to an array of clinical experiences in
hospital and ambulatory settings during their clinical clerkships. Monitoring
these experiences is essential to ensure that students acquire an appropriate
mix of clinical experiences. Attempts made over the last 25 years1 to document the clinical experiences of students have
used such recording devices as casebooks,2
optical scan forms,9,10 handheld
or palmtop computers,9,11 and
pocket-sized encounter cards.12 These studies
have been limited to small samples in isolated clerkships over brief time
and they have largely ignored the severity of illness.
Dramatic changes in the financing and delivery of health care during
the past decade have altered the clinical environments in which medical education
As medical education becomes increasingly decentralized, clinical education
has partially shifted from a tertiary inpatient setting to community-based
and ambulatory sites.3,29- 31
In this kind of educational environment, it is still essential that medical
students encounter a variety of disease entities and are given the opportunity
to perform basic diagnostic and therapeutic maneuvers.
Effective curriculum management requires a valid and reliable system
to document the range and type of students' clinical experiences. Only by
monitoring students' opportunities for clinical encounters with a diverse
mix of patients can informed decisions be made regarding the appropriateness
of a teaching network, training sites, and the balance between inpatient and
ambulatory activities. As medical schools review their learning objectives
to better define the competencies needed by future physicians,32
it will become even more important to document the clinical educational opportunities
offered to medical students.
We investigated the applications of a new system to document key aspects
of the clinical experiences of third-year medical students in 3 clerkships
(family medicine, pediatrics, and internal medicine) at Jefferson Medical
College, Philadelphia, Pa. We provide evidence of validity and reliability,
as well as representative examples of the information produced by the system.
Jefferson's network of affiliated clinical sites includes urban, suburban,
and rural locations throughout Pennsylvania, Delaware, and New Jersey, providing
training in family medicine (6 sites), pediatrics (5 sites), and internal
medicine (8 sites). Family medicine is a 6-week, predominantly ambulatory
rotation; pediatrics is a 6-week rotation including a mix of inpatient and
ambulatory experiences; and internal medicine is a 12-week, predominantly
This project is a collaborative effort involving Jefferson Medical College's
Center for Research in Medical Education and Health Care, the Office of the
Dean, and clerkship directors. The data collection system was developed first
for the family medicine clerkship and then adapted for use in pediatrics and
internal medicine. Data collection began in 1993 for family medicine, in 1994
for pediatrics, and in 1997 for internal medicine. The key issues addressed
by the Clinical Encounter Project include the location of encounters, case
and severity mix of patients, procedures and activities performed by students,
and students' workloads.
At the start of each clerkship students are provided with packets of
5 × 8-inch computer-read patient encounter cards and a pocket-sized
instruction booklet. While the general structure of the cards and booklets
is standard, each department has made modifications to reflect its mix of
patients. In addition to instructions for completing the cards, the booklet
contains a diagnosis list using a small subset of codes from International Classification of Diseases, Ninth Revision, Clinical Modification.33 These codes are used by students
to translate the diagnoses that they write down on the card into numerical
codes for purposes of data analyses. Also in the booklet are examples of the
logic and application of Disease Staging for classifying illness severity.34,35 Disease Staging is a clinically based
classification system with more than 400 disease categories that includes
disease etiology, organ system involvement, and severity of complications.
Severity for specific medical problems is defined in relation to the risk
of organ failure or death. The classification separates each disease into
3 stages based on the severity of its pathophysiologic manifestations. Stage
1 defines a disease with no complications (eg, appendicitis without complications);
stage 2 is a disease with local complications (eg, pneumococcal pneumonia
with empyema); stage 3 is a disease with multiple site involvement or systemic
complications (eg, asthma with respiratory failure).
Students were instructed to complete a card each time they participated
in the care of a patient from whom they obtained a history or performed a
physical examination. For in-hospital locations (internal medicine and part
of pediatrics), multiple encounters with the same patient were recorded on
the same card. Students were instructed to encode each patient's age and sex,
location of encounter, level of involvement and supervision, and diagnostic
procedures that were performed or observed. Principal diagnosis with severity
of illness was recorded, with space provided for up to 4 secondary diagnoses.
Cards were completed at the end of patient care sessions as charting was completed,
or from the students' patient care notes. They were returned and scanned weekly.
Summary reports are prepared twice a year for each department. These
reports are reviewed by the clerkship directors and reviewed annually at departmental
affiliations meetings. These meetings, as well as the usefulness of the data
to the faculty themselves, help to ensure the integrity of the system. Individual
reports, with peer group comparisons, are available to the students.
The total cohort included 647 third-year students (98% of the total
classes) at Jefferson Medical College who had completed clerkships in either
family medicine, pediatrics, or internal medicine in academic years 1997 to
1998, 1998 to 1999, and 1999 to 2000 at Thomas Jefferson University Hospital
and 16 affiliated clinical sites. The remaining 2% included those who either
failed to properly complete or turn in their cards.
We included in our analyses those students who completed at least 30
cards in family medicine, 26 in pediatrics, or 18 in internal medicine. These
criteria were set to represent 75% of the completed cards expected to be received
from each student based on clerkship directors' judgment after 1 to 2 years
of experience with the project (the threshold was lower for internal medicine
due to multiple encounters with the same patient using a single card). While
647 students met the inclusion criteria in at least 1 clerkship, the sample
size for each clerkship ranged from 582 (90%) in family medicine, to 531 (82%)
in internal medicine, to 469 (72%) in pediatrics.
Frequencies and percentages of diagnostic categories and disease severity
were compared using the χ2 test for the significance of the
association between clerkships and severity of diseases for each diagnostic
category. The z test for proportions was used for
pairwise comparisons of proportions for each stage of the diagnostic categories.
Statistical analyses were performed using SAS version 6.12 (SAS Institute,
Students returned 86 011 cards (mean, 133 cards per student): 48 367
in family medicine, 22 604 cards representing 48 799 encounters
in pediatrics, and 15 040 cards representing 90 240 encounters in
internal medicine, or a total of 187 406 encounters. The mean number
of encounters per student was 83 in family medicine, 104 in pediatrics, and
170 in internal medicine. The mean number of encounters for internal medicine
is larger because the clerkship spans 12 weeks, while the others span only
6 weeks. Family medicine and pediatrics are primarily ambulatory clerkships,
having, respectively, 86% and 66% of patient encounters in office environments.
Students in internal medicine reported having more than two thirds of their
encounters in a hospital and 30% in an emergency department.
Geriatric patients comprised 50% of encounters in internal medicine
and 22% in family medicine, while approximately 20% of the family medicine
encounters were with either children or adolescents. The proportions of females
encountered were 60%, 47%, and 45% in family medicine, pediatrics, and internal
The percentages of students who reported encountering at least 1 patient
in each clerkship having 1 of the 10 most frequently encountered principal
and secondary diagnoses are displayed in Figure 1. Differences among the clerkships primarily reflected the
locations of encounters and the ages of patients. The most frequently encountered
principal diagnoses in family medicine included essential hypertension, diabetes
mellitus, back strain, depression, and sinusitis. Asthma, otitis media, diarrhea/gastroenteritis,
and upper respiratory tract infection were the most frequently encountered
diagnoses in pediatrics. Pneumonia and cardiac diagnoses (eg, heart failure,
arrhythmias, acute myocardial infarction) were the most frequently encountered
principal diagnoses in the predominantly inpatient internal medicine clerkship.
Less than half of the students saw any patients with the common medical
problems that have high prevalence rates as reported by the Centers for Disease
Control and Prevention.36 In pediatrics, for
instance, less than half of the students saw a patient with a primary diagnosis
of conjunctivitis (data not shown); in internal medicine fewer than half encountered
syncope as a primary diagnosis. Peptic ulcer disease is another common condition
that was encountered infrequently. Only 11% of students encountered any patients
with this condition during family medicine clerkships; 6% encountered it during
Students were instructed to record multiple diagnoses or medical problems
for each patient, when appropriate. Figure
1 also displays data for these secondary diagnoses. For example,
42% of students reported encountering a patient with a primary diagnosis of
diabetes mellitus during their internal medicine clerkship, and 88% reported
seeing a patient with diabetes mellitus as either the primary or secondary
Data on severity of disease are reported in Table 1 for family medicine and internal medicine rotations. Although
the rate of encountering those disease categories was similar in the 2 clerkships,
severity of the same diseases that students encountered varied. We report
data for family medicine and internal medicine because of the compatibility
For example, of 400 students in family medicine who had seen patients
with diabetes mellitus, 281 (70%) encountered patients in stage 1 of this
disease, while in internal medicine only 50 of 193 (26%) saw such patients.
However, the converse was true for stage 3 of diabetes mellitus; 58% of students
in the internal medicine clerkship, compared with 35% in family medicine,
reported encountering a patient in this stage of the disease. Associations
between severity of disease encountered in different clerkships were statistically
significant by χ2 test. With the exception of stage 2 essential
hypertension, all other differences in percentages across clerkships were
statistically significant by z test for proportions.
Data reported in Table 1 indicate
that during the family medicine clerkship, students were more likely to encounter
early stages of disease and less likely to encounter advanced stages. The
opposite was found for internal medicine.
Each department instructed students to track diagnostic and therapeutic
tasks, including components of the history and physical examination, special
activities such as performance of a Mini-Mental Status Examination, procedures
that the students might be expected to perform in a given clerkship (eg, venipuncture),
as well as procedures they might observe (eg, sigmoidoscopy). The number and
percentage of students who performed the most frequently reported procedures
in each clerkship, as well as the mean and median number performed per student,
are shown in Table 2. For example,
84% (489/582) of students in family medicine clerkships and 82% (385/469)
in pediatrics clerkships performed breast examinations, while 37% (198/531)
in internal medicine did so. In family medicine, 81% (472/582) reported doing
a pelvic examination. None performed this procedure in pediatrics, and only
13% (67/531) performed the procedure in internal medicine. Health promotion/counseling
activities were performed by 91% (532/582) of the students in family medicine,
93% (436/469) in pediatrics, and 77% (410/531) in internal medicine clerkships.
On average, each student performed 9 breast examinations (median, 6),
and offered health promotion or counseling 38 times (median, 28) during the
3 clerkships. Some procedures were observed by students more often than performed.
For example, while 40% of students reported performing an electrocardiogram
in either family medicine or internal medicine, 80% reported observing at
least 1 electrocardiographic procedure (data not shown).
To summarize experiences across clerkships, we analyzed the data for
358 students who met our inclusion criteria in all 3 clerkships (Table 2). For example, 13% of students
in family medicine, 22% in pediatrics, and 38% in internal medicine reported
administering a purified protein derivative (PPD) skin test for tuberulosis
(Table 2). However, when we examined
across all 3 clerkships, 55% of students reported administering a PPD.
To assess the validity of students' diagnoses, we selected 3 days randomly
and examined 112 encounter cards completed by 15 students in the family medicine
clerkship. Three of the authors (D.Z.L., C.R., F.W.M.) reviewed charts set
aside from office sessions. The reviewers compared the attending faculty's
principal diagnosis from the chart to the diagnosis on the previously submitted
cards. Concordance was 77% between the students' recorded principal diagnoses
and the faculty's confirmation of diagnosis. The concordance rate increased
to 97% when either principal or secondary diagnoses were considered. These
concordance rates are much higher than those reported in a psychiatry clerkship
(range, 33%-49%).8 Our concordance rates are
comparable with those reported in a primary care clerkship (88% and 87%).24 Findings reported in Figure 1, regarding the most prevalent diseases in each clerkship,
provide further support for the validity of the system (eg, concordance between
expecting and actually encountering a large number of patients with hypertension
in family medicine, otitis media in pediatrics, and pneumonia in internal
medicine clerkships). The expected variation in disease severity (Table 1) reflects the settings in which
encounters were recorded (outpatient, therefore less disease severity in family
medicine; inpatient, more disease severity in internal medicine), providing
further validation evidence.
To support the reliability of this documentation system we compared
the pattern of diagnostic categories for each clerkship in different years.
Similar patterns of diagnostic encounters were found for each clerkship across
the 3 academic years (1997-1998, 1998-1999, 1999-2000), indicating that the
pattern of patient encounters remained stable over time, supporting the reproducibility
of the results. For example, in pediatrics, 8 of the 10 most frequently reported
diagnoses/problems were on the list in all 3 years.
Previous studies of student encounters with patients have focused on
a single discipline such as family medicine,3,9,13- 17
internal medicine,18- 20
or psychiatry,8 or on the ambulatory setting.21,22,24 Exceptions are a
comparison of the encounters of 40 students in family medicine to those in
other clerkships,23 and a study of 17 918
patient encounters during family medicine and internal medicine rotations
at 2 medical schools for 1 year.37 Our study
reports on a system that has been in place for 8 years and that assesses 3
major clerkships in terms of patient disease severity.
Although the present study focused on only 3 clerkships, we believe
that this methodology could be applied to any clerkship that involves encounters
with patients. In our educational network this system has been initiated in
the third-year clerkships in surgery and obstetrics/gynecology and will soon
begin in psychiatry. Objective data on types of clinical experiences could
be used to identify curriculum deficiencies, to design supplementary experiences,
and to assess the results of interventions designed to enhance various aspects
of clinical encounters.
Focused information about the clinical experiences of each medical student
can be used in the planning, implementation, and evaluation of educational
experiences as well as in counseling individual students. Trend data are essential
in the ongoing evaluation of the suitability and effectiveness of each educational
site. Data collected by this documentation system can also identify the types
of patient problems that students do not encounter and procedures they do
not perform. To address clerkship deficiencies and monitor a student's progress,
the student could be asked to evaluate a patient with critical problems and
perform some of these procedures in the same or other clerkships.
For instance, in an earlier study we noted that in a family medicine
clerkship, female students were not performing male genital examinations as
frequently as were their male peers.38 By monitoring
clinical experiences we were able to correct this deficiency and improve female
students' participation. Similarly, in pediatrics, few students reported doing
such office procedures as screening patients' hearing and vision. A checklist
is now in place at all training sites to remind students and faculty to include
these procedures in students' clinical experiences.
Experiences within both inpatient and outpatient educational settings
are essential to expose each student to a diversity of common medical problems
and to a spectrum of illness severity within specified diagnostic categories.
Interviewing a patient with diabetes as an outpatient to address glycemic
control and tertiary prevention is a very different educational experience
from that of caring for a patient admitted to the hospital for management
of diabetic ketoacidosis. It also differs from the educational experience
of caring for a patient with a myocardial infarction who also has diabetes.
Recording multiple problems highlights the educational importance of each
In summary, this surveillance system provides insight into disease frequency
and the diversity of case mix and has the unique feature of taking into account
the severity of medical problems that students encounter. Monitoring students'
educational opportunities in different clerkships and at different educational
sites is necessary if educators are to understand and optimize those clinical