Tamblyn R, Abrahamowicz M, Brailovsky C, Grand'Maison P, Lescop J, Norcini J, Girard N, Haggerty J. Association Between Licensing Examination Scores and Resource Use and Quality of Care in Primary Care Practice. JAMA. 1998;280(11):989-996. doi:10.1001/jama.280.11.989
From the Departments of Medicine (Dr Tamblyn) and Epidemiology and Biostatistics (Drs Tamblyn and Abrahamowicz and Mss Girard and Haggerty), McGill University, Montreal, QuÉbec; the Centre d'Évaluation des Sciences de la SantÉ, University of Laval, QuÉbec City, QuÉbec (Dr Brailovsky); the Department of Family Medicine, University of Sherbrooke, Sherbrooke, QuÉbec (Dr Grand'Maison); the QuÉbec College of Physicians, Montreal, QuÉbec (Dr Lescop); and the American Board of Internal Medicine, Philadelphia, Pa (Dr Norcini).
Context.— Clinical competence is a determinant of the quality of care delivered,
and may be associated with use of health care resources by primary care physicians.
Clinical competence is assumed to be assessed by licensing examinations, yet
there is a paucity of information on whether scores achieved predict subsequent
Objective.— To determine if licensing examination scores were associated with selected
aspects of quality of care and resource use in initial primary care practice.
Design.— Prospective cohort study of recently licensed family physicians, followed
up for the first 18 months of practice.
Setting.— The Québec health care system.
Participants.— A total of 614 family physicians who passed the licensing examination
between 1991 and 1993 and entered fee-for-service practice in Québec.
Main Outcome Measures.— All patients seen by physicians were identified by the universal health
insurance board and all health services provided to these patients were retrieved
for the 18 months prior to (baseline) and after (follow-up) the physicians'
entry into practice. Medical service and prescription claims files were used
to measure rates of resource use (specialty consultation, symptom-relief prescribing
compared with disease-specific prescribing) and quality of care (inappropriate
prescribing, mammography screening). Baseline data were used to adjust for
differences in practice population.
Results.— Study physicians saw a total of 1116389 patients, of whom 113535 (10.2%)
were elderly and 83391 (7.5%) were women aged 50 to 69 years. Physicians with
higher licensing examination scores referred more of their patients for consultation
(3.8/1000 patients per SD increase in score; 95% confidence interval [CI],
1.2-7.0; P=.005), prescribed to elderly patients
fewer inappropriate medications (−2.7/1000 patients per SD increase
in score; 95% CI, −4.8 to −0.7; P =.009)
and more disease-specific medications relative to symptom-relief medications
(3.9/1000 patients per SD increase in score; 95% CI, 0.3 to 7.4; P=.03), and referred more women aged 50 to 69 years (6.6/1000 patients
per SD increase in score; 95% CI, 1.2-11.9; P=.02)
for mammography screening. If patients of physicians with the lowest scores
had experienced the same rates of consultation, prescribing, and screening
as patients of physicians with the highest scores, an additional 3027 patients
would have been referred, 179 fewer elderly patients would have been prescribed
symptom-relief medication, 912 more elderly patients would have been prescribed
disease-specific medication, 189 fewer patients would have received inappropriate
medication, and 121 more women would have received mammography screening.
Conclusions.— Licensing examination scores are significant predictors of consultation,
prescribing, and mammography screening rates in initial primary care practice.
PRIMARY CARE physicians are the foundation of westernized health care.
They manage episodic and chronic illness, play a central role in instituting
interventions for primary and secondary disease prevention, prescribe and
monitor the majority of drug treatment, and function as the entry point to
costly health care resources.1- 7
The quality of care delivered by primary care physicians and their use of
resources can have a considerable impact on the cost-effectiveness of health
care delivery.3 Preventive care,8- 11
and chronic disease management21- 28
have been identified as areas in which quality of care could be improved.
Continuing medical education has been identified as a solution to this
problem because low self-reported rates of preventive care are associated
with inadequate knowledge of preventive guidelines,29,30
and improvements in prescribing and chronic disease management have been reported
with disease-specific education in diagnosis and management.31,32
Yet mandatory continuing medical education requirements have not been associated
with improvements in practice,33,34
and this observation has likely contributed to a call for mandatory assessment
of the competence of practicing physicians through relicensure and recertification
The question that has not been adequately addressed is whether tests
that purport to measure clinical competence (licensing and certifying examinations)
actually predict performance in subsequent practice. Ramsey et al,37 who were the first investigators to address this
question, found that scores on the American Board of Internal Medicine certification
examination predicted colleagues' ratings of the quality of care delivered
by internists 5 to 8 years after certification. Although these results are
promising, we do not know whether they are generalizable to family physicians
and general practitioners, physicians who provide the majority of primary
care, or if colleagues' ratings actually predict what is done in practice.
The possibility that licensing examination scores may predict performance
in practice needs to be examined because interventions to improve practice
through licensure and certification provide the ultimate opportunity for an
ounce of prevention to be worth a pound of cure.
Clinical competence has not only been considered to be relevant to the
quality of care delivered, but it has also been associated with variation
in the use of resources by primary care physicians. Two studies, one British38 and one American,39
have found that primary care physicians with higher self-reported competence
were more likely to refer their patients for specialty consultation. This
is relevant because referral is the main mechanism by which patients gain
access to hospital resources and expensive surgical and investigative procedures,3 and unexplained differences in referral rates exist
among primary care physicians.40- 43
Considerable variation also exists among primary care physicians in prescription
and it has been suspected that these differences are related to clinical competence.50,51 In particular, higher rates of prescription
of medications often used for the relief of undifferentiated symptoms49,52- 60
in the elderly (eg, benzodiazepines, nonsteroidal anti-inflammatory drugs
[NSAIDs], analgesics) are seen among physicians who have low rates of prescribing
for disease-specific treatments (eg, hypoglycemics, cardiac medications).46,47 It has been postulated that this
pattern is related to an underdiagnosis and undertreatment of disease because
of poorer skills in diagnosis and management. There has been no direct test
of this association.
Several methodological problems have limited research on the relationship
between licensing and certification examination scores and practice. First,
physicians with higher scores are more likely to participate in follow-up
studies.61 Second, mandatory examinations have
traditionally been completed at the end of medical school, with only selected
groups of physicians going on to take voluntary certification examinations.
Thus, to include all physicians, evaluation would have to take place at a
point in training when competence in diagnosis and management would not be
fully developed. Finally, follow-up studies of physicians into practice are
expensive because assessment of 50 to 200 patients may be needed to obtain
stable estimates of performance, and the alternate of self-reported activity
is not a valid assessment of what physicians actually do in practice.62,63
An opportunity to overcome 4 major methodological difficulties existed
in Québec, a Canadian province with a population of 7.2 million people,
4 medical schools, and approximately 17000 physicians. First, in this province,
mandatory assessment of a physician's competence is carried out shortly before
an individual enters practice. To be eligible for licensure, physicians must
complete a family medicine or other specialty residency and then pass a combination
licensing and specialty certification examination. Second, physicians are
tested in specific areas of competence and must meet minimum standards in
each area to be eligible for licensure. As a result, the subset of competencies
that would be expected to have the strongest link with specific practice activities
could be assessed. In addition, Québec was the first jurisdiction to
include an objective structured clinical examination (OSCE) in its licensure
examination process.64 This provided the opportunity
to evaluate the utility of this new form of testing of clinical skills. Third,
all medical services provided to provincial residents are covered by a universal
health insurance plan with 1 government agency responsible for payment of
all physician services. This provided the opportunity to obtain a complete
census of the practice activity of fee-for-service physicians. Fourth, the
availability of high security encryption technology and third-party linkage
provided methods of protecting the confidentiality of individual physicians
so that all physicians could be included in a study. Although differences
in licensing examination requirements and medical service data acquisition
exist in Québec, undergraduate education and family medicine training
are similar to other accredited North American programs.
In the current study, we used this opportunity to evaluate several aspects
of primary care in which variation in care had been identified and in which
practice activity could be measured using information retained in provincial
health care databases. In relationship to quality of care, we tested the hypotheses
that better clinical skills and prevention knowledge would be related to higher
rates of mammography screening and that higher scores in diagnosis and management
would be related to lower rates of potentially inappropriate prescribing.
We included the assessment of clinical skills as a potential predictor of
mammography screening because of the reported association between physical
examination and preventive care.65 In relationship
to resource use, we tested the hypotheses that higher overall competence would
be related to higher consultation rates and that higher scores in diagnosis
and management would be related to lower rates of symptom-relief prescribing
relative to disease-specific prescribing.
A prospective follow-up study of a cohort of newly licensed family physicians
was conducted. Physicians were eligible for inclusion in the study if they
had passed the family medicine licensing examination in 1991, 1992, or 1993;
applied for a license to practice in Québec; and entered fee-for-service
practice. Physicians paid by salary were excluded because there was no accurate
way to identify all patients who had been seen by these physicians even though
all prescriptions and referrals for consultation and mammography could be
ascertained. Potentially eligible physicians were identified by the Québec
College of Physicians, Montreal, Québec, and the license numbers of
these physicians were provided to the provincial health insurance agency (Régie
de l'Assurance-Maladie du Québec [RAMQ], Québec City, Québec).
Practice activity was assessed in the 18-month period after the earliest possible
entry into practice. For each cohort, the follow-up period was between July
1 of the examination year and December 31 of the following year.
Candidates applying for licensure in Québec must pass the College
of Family Physicians of Canada Certification Examination (CFPCEx)
and the Québec OSCE.64,66
The CFPCEx assesses 4 areas of competence: diagnosis, management,
prevention, and communication.66 Diagnosis,
management, and prevention are measured by multiple-choice questions and short-answer
management problems and communication is measured by a simulated patient office
oral examination. To pass the CFPCEx, a candidate must obtain a
score of at least 60% each in diagnosis, management, and communication and
a score of at least 50% in prevention. The OSCE primarily measures clinical
skills by direct, structured observation of a candidate's performance in 24
to 26 clinical problems.64 Clinical problems
are presented by standardized patients and performance is assessed by physician
examiners. Scores for each case are averaged to produce an overall OSCE score.
Candidates fail the OSCE if their score is more than 2 SDs below the mean.
Candidates who pass some but not all components of the 2 examinations are
reviewed by the licensing examination board and decisions are made on a case-by-case
The subscores from the CFPCEx and the OSCE and an overall
average score (the mean of all scores) were retrieved for eligible physicians.
There were 6 administrations of the licensing examination between 1991 and
1993. To adjust for potential differences in the difficulty of the examination,
scores were standardized using the reference-group method.67
A large homogeneous group of examinees (first-time takers from North American
medical schools) is assumed to have similar ability over time and this group
was used as the reference standard. Reliability of scores was estimated in
each year and summarized across years using a weighted intraclass correlation
coefficient.68 Standardized examination scores
for each physician and their license number were sent to the RAMQ. Personal
identifiers on the score file and health care databases were encrypted to
protect confidentiality but permit database linkage.
Data Source. Practice activity and practice population characteristics were assessed
using information from 4 provincial health care databases and the national
1992 census data. The physician claims file provides, for all medical services
delivered on a fee-for-service basis (95% of services69
), information on the treating physician; referring physician; recipient;
type, location, and date of service; and the diagnosis for the visit. The
prescription claims file provides information on all prescription drugs dispensed
to the elderly, including the recipient, prescribing physician, drug, quantity
dispensed, and dispensing date. These reimbursement files are systematically
audited and have been independently validated.70,71
The registrant database contains the registration number, age, sex, residential
address, and health district of each provincial resident. The master physician
file contains information on the license number, specialty, year of graduation,
medical school, age, and sex of Québec physicians. The national 1991
census data72 were linked to the registrant
database to provide information on the average family income and mean educational
achievement in the health district of each registrant.
Data Retrieval. A 3-step process was used to obtain practice data for each physician.
First, the RAMQ identified all registrants who had been seen on a fee-for-service
basis by each study physician in the 18 months following their earliest possible
entry date into practice. Second, all services and prescriptions provided
to these patients by the study physician as well as other physicians in the
province were retrieved from the 18-month follow-up window established for
each physician. Third, all services and prescriptions provided to these patients
were retrieved from the 18-month period before the follow-up window. These
baseline data were used to account for differences in the practice population
characteristics that existed prior to the first contact with the study physicians
and, thus, could not be attributed to their practice patterns.
Practice activity for each physician was measured in 3 practice populations:
all outpatients, elderly patients, and women aged 50 to 69 years. Patients
were considered to be in a physician's practice population if they received
1 or more services from the study physician in an outpatient setting (emergency
department, outpatient clinic, or office practice). For women to be considered
eligible for mammography screening, we further restricted the practice population
to women seen in an outpatient setting where the institution of preventive
care would normally be expected. The setting in which care was delivered was
determined by the location recorded on the billing claim, and age restrictions,
when relevant, were determined by the age recorded for an individual in the
registrant database. Each practice population denominator was composed of
the number of different patients seen by a physician during the follow-up
period. Patients who were seen by multiple study physicians were represented
in more than 1 physician's practice population. Four practice indicators were
generated for each physician: consultation rate, symptom-relief prescribing
rate compared with disease-specific prescribing rate, inappropriate prescribing
rate, and mammography screening rate in women aged 50 to 69 years.
Consultation rate was defined as the proportion
of outpatients in a physician's practice who were referred at least once for
a specialty consultation by the study physician. A patient was designated
as having been referred to the specialist when the study physician was identified
by the specialist as the referring physician on the claim for a consultation.
Symptom-relief prescribing rate relative to disease-specific
prescribing rate was defined as the difference between the proportion
of elderly patients for whom the study physician prescribed disease-specific
medication and the proportion of elderly patients who were prescribed symptomatic
medication by the study physician. Symptomatic medication, defined as drugs
that relieve symptoms but have little impact on the disease process,47 included NSAIDs, benzodiazepines, and low-dose narcotic
analgesics. Disease-specific medication, defined as drugs that would likely
be used exclusively to treat an investigation-confirmed disease state,47 included anticoagulants, anticonvulsants, antidepressants,
antihypertensives, anti–Parkinson disease medication, asthma preparations,
corticosteroids, diuretics, antiglaucoma medication, cardiac medication, hypoglycemic
medication, and thyroid and antithyroid medication. (The list of medications
included in symptom-relief and disease-specific prescribing rates is available
on request from the authors.) The list of disease-specific and symptomatic
medications was created using the classification established by McGavock et
al,47 excluding drugs that could be obtained
as over-the-counter products. The prescription of drugs started by other physicians
and refilled by the study physician was distinguished from the prescription
of drugs started by the study physician (no evidence of a previous prescription
for the drug in the 6 months preceding the study physician's prescription)
for both disease-specific and symptom-relief medication.
Inappropriate prescribing rate was defined
as the proportion of patients prescribed a relatively contraindicated psychotropic
drug, cardiovascular drug, or NSAID. Inappropriate drugs were identified from
a published international expert consensus73
and were defined as medications that should be avoided in the elderly because
of the risk of toxic effects (long-acting benzodiazepines, meprobamate, barbiturates,
amitriptyline, indomethacin, phenylbutazone, propoxyphene, pentazocine, reserpine,
methyldopa, and propranolol). Prescriptions of inappropriate drugs that were
started by other physicians and refilled by the study physician were distinguished
from drug prescriptions that were started by the study physician without evidence
of a previous prescription in the past 6 months.
Mammography screening rate in women aged 50 to 69
years was defined as the proportion of eligible women aged 50 to 69
years who received bilateral mammography at the request of the study physician.
Eligible women were those who had no diagnosis of breast cancer or breast
disease or a diagnostic mammogram before their first contact with the study
physician. To assess whether omissions in care were due to the provision of
mammography screening by other physicians, we also determined the overall
rate of screening for each study physician's practice, irrespective of the
For each practice subpopulation (all outpatients, elderly patients,
and women aged 50-69 years), information was first retrieved for each patient
and then aggregated to produce summary measures for the practice (available
from the authors on request). Variables used to summarize the practice population
included age, sex, average income, education, geographic access to health
care, comorbidity, prior health care use, and outcome-specific utilization
of health services. Geographic access was treated as a potential confounder
in comparisons of specialty consultation (distance to an urban center) and
mammography screening rates (distance to a mammography screening facility)
and practice population characteristics and outcome-specific utilization behavior
as confounders for consultation, mammography, and symptom-relief prescribing
rates. For inappropriate prescribing, no confounders were considered relevant
because the decision to prescribe a relatively contraindicated medication
is rarely justified by patient characteristics.
Multiple linear regression was used to test hypothesized associations
between examination subscores and practice outcomes when adjusted for potential
confounders. To account for differences in the precision of the estimated
outcome rates for each physician, observations were weighted by the square
root of the number of eligible patients in each physician's practice. The
residuals were examined to assess whether a linear model was appropriate.
All practice population characteristics were included in the regression models
for consultation rate, symptom-relief prescribing rate compared with disease-specific
prescribing rate, and mammography screening rate to control for potential
confounding due to differences in the composition of the practice population.
Because physicians entered practice at different points in the 18-month period,
we controlled for varying amounts of follow-up by including months of active
practice as a continuous covariate in the regression models. Each examination
score was modeled separately to avoid problems of multicollinearity in the
estimation of the regression coefficients. The population impact of variation
in examination scores was determined based on the differences in the rate
of each outcome among patients seen by physicians at the upper (≥2 SDs
above the mean) and lower (≥2 SDs below the mean) end of the score distribution.
We calculated the difference in the number of outcomes that would be expected
among patients seen by physicians with lowest scores if they experienced the
same outcome rate as patients of physicians with the highest scores.
A total of 810 family medicine residents took the licensing examination
in 1991, 1992, or 1993, and 614 were eligible for inclusion. Among the ineligible
residents, 21 failed, 46 did not apply for Québec license, 17 had not
started practice within the first 18 months, and 112 entered salaried practice.
Of the 614 physicians included in the analysis, 56.0% were female, 85.8% were
graduates from Québec medical schools, and 6.5% were foreign medical
graduates. Most (90%) had graduated between 1989 and 1991; 36.5% took the
examination in 1991, 36.6% in 1992, and 26.9% in 1993.
Average standardized examination scores for the 614 physicians were
slightly below the mean of 0 for the reference group (Table 1). This is because the reference group had, on average, higher
scores than the entire group of physicians who passed the examination, and
they were relatively homogeneous in their performance. The percentage of study
physicians who were 3 or more SDs below the reference group mean varied from
0.8% (OSCE) to 3.1% (prevention). For prevention, the average score for the
reference group varied from 74% to 80% in different administrations (SD, 7.5%-7.6%).
Thus, physicians who obtained the passing score of 50% for prevention would
have a standardized score of –2.8 to –3.9 depending on the year
in which they took the examination. Score reliability varied from 0.27 to
0.72. Correlations among examination scores varied from a low of r=0.36 for prevention and OSCE scores to a high of r=0.63 for diagnosis and management scores.
During the 18-month follow-up period, the 614 study physicians saw a
total of 1116389 individuals, 16% of the total population in the province.
Seventeen percent of individuals were seen by more than 1 study physician.
Over 74 million claims for medical services and prescriptions were retrieved
for these patients, 31574888 from the baseline period and 42850337 from the
follow-up period. Physicians practiced in an average of 3.6 settings (median,
4; range, 1-7) in the first 18 months and the distribution by geographic location
was 71.8% in urban or periurban areas, 25.4% in rural and remote settings,
and 2.8% in mixed locations. The average outpatient practice population size
was 2250 patients per study physician, of which 251 were eligible elderly
patients and 161 were eligible women aged 50 to 69 years. The average (SD)
length of follow-up was 15.8 (3.2) months.
During the follow-up period, the average consultation rate in the practices
of study physicians was 71 per 1000 patients (SD, 51). On average, 108 elderly
patients per 1000 (SD, 109; range, 0-563) were prescribed disease-specific
medication and 126 elderly patients per 1000 (SD, 103) were prescribed symptom-relief
medication, the most common drug being a benzodiazepine (54/1000). The average
difference in prescribing rates was negative (mean [SD], −18 /1000;
range, −167 to 188), indicating that physicians prescribed symptom-relief
drugs to proportionately more elderly patients than they prescribed disease-specific
medications. However, the range indicates that differences in disease-specific
and symptom-relief prescribing rates varied considerably among physicians.
Seventy-one percent of study physicians prescribed an inappropriate drug to
at least 1 of their elderly patients. The average rate of inappropriate prescribing
across practices was 20 per 1000 elderly patients (SD, 31; range, 0-310).
Approximately half (9/1000) of all patients who were prescribed an inappropriate
medication were started on the drug therapy by the study physician, the most
common drug being a long-acting benzodiazepine. The mammography screening
rate was 51 per 1000 women aged 50 to 69 years (SD, 81; range, 0-750) when
restricted to mammograms requested by the study physician and 218 per 1000
when all screening mammograms ordered for study physicians' patients were
Higher consultation rates were associated with higher overall examination
scores (Table 2). For every SD
increase in a physician's overall competence score, 3.8 additional patients
per 1000 would be referred for consultation.
Physicians with higher diagnostic and management scores were more likely
to have higher rates of disease-specific prescribing relative to symptom-relief
prescribing and were less likely to prescribe symptom-relief medication or
inappropriate medication. With every SD increase in diagnostic score, 3.7
more elderly patients per 1000 would be prescribed disease-specific medication
relative to symptom-relief medication, 6.5 fewer elderly patients per 1000
would be prescribed symptom-relief medication and 2.7 fewer elderly patients
per 1000 would be prescribed potentially inappropriate medication. Higher
management scores were not significantly associated with symptom-relief prescribing
but were associated with higher relative rates of disease-specific to symptom-relief
prescribing and lower rates of potentially inappropriate prescribing. The
regression coefficients were similar when the analysis was limited to treatment
that was started by the physician. To determine if the exclusion of salaried
physicians biased our estimates, we evaluated the rate of inappropriate prescribing
among elderly patients prescribed a drug by the study physician. Diagnostic
score was the most significant predictor in models that included and excluded
salaried physicians and the estimated regression coefficient was the same
in both models (β, −5/1000; P =.01).
Higher rates of mammography screening were associated with higher scores
in prevention and clinical assessment. For every SD increase in these scores,
we estimated that an additional 5.4 to 6.6 women per 1000 would be screened
for breast cancer. The inclusion of mammograms ordered by other physicians
tended to lower the estimated regression coefficients for each of the examination
Although a relatively small number of physicians obtained subscores
that were 2 or more SDs below (7.2%-9.9%) or above (4.7%-6.5%) the mean, these
physicians saw a considerable number of patients in the initial practice period.
If patients of physicians with the lowest scores experienced the same rates
of consultation, prescribing, and screening as patients of physicians with
the highest scores, an additional 3027 of the 125383 outpatients seen by the
lowest-scoring physicians would have been referred. Of the 9718 elderly patients
seen, 179 fewer would have been prescribed symptom-relief medication, 912
more would have been prescribed disease-specific medication, and 189 fewer
would have received inappropriate medication. For the 5335 women aged 50 to
69 years who were seen, 121 more would have received mammography screening.
In this study, we found that clinical competence, assessed by a combined
licensing and certifying examination at the end of training, predicted consultation,
prescribing, and mammography screening in initial primary care practice. The
strengths of this study were that we were able to study all physicians who
entered fee-for-service practice, assess their clinical competence at the
end of training with state-of-the-art methods of assessment, and capture information
on all patients seen by these physicians in the initial practice period. Both
the strengths and main limitations of the study relate to the use of secondary
databases for outcome assessment. On the positive side, we were able to avoid
biases related to self-selection, self-report, and poor documentation. On
the negative side, we could not measure practice organization or many outcomes
that would be of relevance in assessing the adequacy of primary care delivery
such as episodic and chronic disease management and patient satisfaction.
The relationship of examination scores to these outcomes remains unknown.
As clinical information is limited, we were unable to determine the appropriateness
of resource use or whether extenuating circumstances explained rates of mammography
referral or inappropriate prescribing. Limited precision in classifying outcomes
would have likely led to an underestimation of the strength of the relationship
between examination scores and practice outcomes. Overestimation of the effect
would have been possible only if physicians with lower scores were more likely
to see patients who did not require referral; were more likely to have indications
for benzodiazepines, NSAIDs, or relatively contraindicated therapy; and were
less likely to comply with physician referral for mammography.
This study supports the validity of licensing examination as measures
of clinical competence. This is good news for licensing and certifying bodies
because these examinations have been used to judge whether an individual has
sufficient knowledge, skills, and judgment to deliver safe and effective medical
care, and yet there has been a paucity of information about whether these
assumptions are correct.74 Furthermore, this
study supports the contention that specific competencies are directly related
to the aspects of practice in which these abilities are assumed to be required.
For example, knowledge of prevention was an important predictor of mammography
referral rate. These findings legitimize the approach taken in this examination
to require demonstration of a minimum level of proficiency in each of the
major aspects of clinical competence: diagnosis, management, prevention, and
clinical skills. It also raises questions about the more common practice of
establishing 1 overall passing score since this may mask important weaknesses
in specific domains.
The observation that 71% of newly licensed physicians prescribed potentially
inappropriate medication to their elderly patients, and for 1 physician that
31% of his or her elderly patients were prescribed potentially inappropriate
medication, raises the question of whether the passing standard was high enough.
Methods of establishing the passing scores for examinations have not had empirical
information that could be used to assess the risk of specific deficiencies
in knowledge or ability on the quality of care that would be delivered to
the population or consequences for health outcome. Either a specified proportion
of examinees at the lower tail of the score distribution fail, regardless
of their ability (norm-referenced approaches), or a minimum absolute standard
is established that would be expected to fail about half of theoretically
"borderline" candidates (criterion-referenced approaches). Practice outcome
information such as that produced in this study could be incorporated into
criterion-referenced methods of standard setting, and this may be an important
area for future research.
From the perspective of training programs, there are 2 relevant issues.
First, from the literature we know that the quality of a training program
has been shown to be positively associated with the scores achieved on certification
examinations,75 and furthermore, that certification
examination scores, at least in internal medicine, are associated with quality
of practice as rated by colleagues.37 This
means that improvements in training programs may have an appreciable effect
on the quality of care delivered by graduates. Second, scores on licensing
examinations tend to be highly correlated with scores on the same types of
tests taken during medical school.76- 80
If licensing examination scores predict some aspects of future practice, then
scores achieved during medical school may also be predictive. This supposition
should be tested because undergraduate and postgraduate training programs
have the opportunity to provide remedial programs for deficiencies identified
at a much earlier point in training.
The focus in medical education and licensing examinations has usually
been on the individual physician, not on the impact that a physician will
have on his or her patient population. This study quantified the number of
patients who are influenced by new physicians. Indeed, one sixth of the provincial
population, more than a million patients, received services from these physicians
in the first 1.5 years of practice. Even when risks are small, such as the
case in this study in which only modest associations were evident between
examination scores and practice outcomes, the effects of potentially suboptimal
care delivery by lower-scoring physicians may be appreciable if a substantial
proportion of the population is affected. If this effect persists over time,
the population-attributable risk of receiving treatment from physicians with
lower levels of competence may be considerable. This study could not answer
this question. We do not know if the behavior exhibited in the early period
of practice had an impact on health outcome or if these behaviors persist
over time. Nevertheless, the associations we did observe raise some questions
about current policy and practice.
First, we substantiated the observations of earlier studies40,41; physicians who have higher competency
scores are more likely to refer their patients. What does this mean? We could
assume that high-scoring physicians are overly fastidious in early practice.
Alternately, at a time when health policy has encouraged a reduction in use
of specialty consultation for cost containment, we could and should worry
that physicians who have lower scores may be less likely to recognize the
significance of a patient's problem or their own limitations. If this were
the case, the apparent cost savings associated with low referral rates may
mask poorer quality of care and even greater costs, both human and administrative,
of avoidable morbidity.81
Prescribing activity of physicians has also come under the spotlight
in health policy reforms and drug review assessments.13- 20,82- 84
Annual expenditures for prescription drugs have increased substantially,85- 87 and drug-related
illness is now claimed to be the sixth leading cause of mortality in the United
States.88 The obvious question for both consumers
and payers is whether prescription drugs are being used wisely. This study
showed that drugs that may be overused or relatively contraindicated in the
elderly are more likely to be prescribed by physicians with lower scores in
diagnosis and management. We have no way of knowing whether these prescriptions
were justified, but our results supported the conviction of McGavock et al46,47 that high rates of symptom-relief
prescribing compared with disease-specific prescribing may be more common
in physicians with poorer skills in diagnosis and management. A considerable
investment has been made in interventions to improve the drug-prescribing
habits of practicing physicians.31,56,89
This study provides new insights into this problem, namely, that training
and licensure may provide a new avenue for preventive intervention that ultimately
may be far less costly than remedial interventions for practicing physicians.90
Physician recommendation has been identified as the most important determinant
of the likelihood that a woman will receive a mammogram.91- 94
Optimal levels of screening have not been achieved,6,8- 10,95
and physician knowledge and attitudes have been identified as significant
determinants of their intention to institute screening practices.29,30 To our knowledge, our study provided
the first direct assessment of the association between competence in clinical
assessment and prevention and mammography screening behavior. Our findings
are important because physicians who are more likely to screen for breast
cancer are also more likely to carry out other preventive services.6,96 Changes in training programs and licensing
examinations may provide a means of optimizing preventive care, and this should
be the subject of future research.
Licensing examination scores are significantly associated with some
aspects of initial practice behavior. Changes in the passing standard and
training may provide an appropriate and effective method of optimizing practice.
To determine if this is true, future studies should broaden the scope of practice
activities assessed to include continuity of care, other aspects of preventive
care and counseling, and the quality of episodic and chronic disease management.
The association between examination scores and patient outcomes such as patient
satisfaction, quality of life, and avoidable morbidity and mortality would
also be important to assess. Future research should also determine if the
association between examination scores and practice activity persist over
time and whether examinations taken earlier in training are also predictive
of future practice activity.