Context.— Intracranial hemorrhage must be excluded prior to administration of
thrombolytic agents in acute stroke.
Objective.— To evaluate physician accuracy in cranial computed tomography scan interpretation
for determining eligibility for thrombolytic therapy in acute stroke.
Design.— Administration of randomly selected, randomly ordered series of 15 computed
tomography scans from a pool of 54 scans that demonstrated intracerebral hemorrhage,
acute infarction, intracerebral calcifications (impostor for hemorrhage),
old cerebral infarction (impostor for acute infarction), and normal findings.
Participants.— A convenience sample of 38 emergency physicians, 29 neurologists, and
36 general radiologists.
Main Outcome Measures.— Physician determination of eligibility for thrombolytic therapy based
on computed tomography scan interpretation.
Results.— Average correct score by all physicians on all computed tomography scans
was 77% (95% confidence interval, 74%-80%). Of 569 computed tomography readings
by emergency physicians, 67% were correct; of 435 readings by neurologists,
83% were correct; and of 540 readings by radiologists, 83% were correct. Overall
sensitivity for detecting hemorrhage was 82% (95% confidence interval, 78%-85%);
17% of emergency physicians, 40% of neurologists, and 52% of radiologists
achieved 100% sensitivity for identification of hemorrhage.
Conclusion.— Physicians in this study did not uniformly achieve a level of sensitivity
for identification of intracerebral hemorrhage sufficient to permit safe selection
of candidates for thrombolytic therapy.
RECOMBINANT tissue plasminogen activator has been approved for select
patients within 3 hours of onset of acute ischemic stroke.1
Since thrombolytic therapy may produce lethal bleeding in patients with intracranial
hemorrhage, the presence of intracranial blood on the initial computed tomography
(CT) scan has been an exclusion criterion in the 5 trials of thrombolytic
therapy for stroke2-6
and in the recommendations of expert panels.7-9
Thus, before thrombolytic therapy may be given in stroke, a physician highly
skilled in identifying intracranial hemorrhage must interpret the CT scan.
Early signs of major cerebral infarction (sulcal effacement, mass effect,
and edema) also are associated with an increased risk for intracerebral hemorrhage
in patients who receive thrombolytic therapy,10-12
and some guidelines recommend avoiding thrombolytic therapy when these findings
are present.7-9
We conducted this study to determine how well emergency physicians,
neurologists, and general radiologists identified cranial CT scans that have
evidence of intracranial hemorrhage. We also assessed these physicians' accuracy
in interpreting other CT findings commonly seen in patients with acute ischemic
stroke to examine whether physicians could distinguish CT scans that demonstrate
subtle imaging abnormalities representing potential contraindications to thrombolytics
from those that do not preclude safe administration of these agents.
Development of Scan Library
We reviewed interpretations of all cranial CT scans performed at a university
teaching hospital from December 1994 to January 1996 to identify scans exhibiting
hemispheric parenchymal hemorrhage or early infarction. Scans with calcifications
(used as impostors for hemorrhage), scans with old infarction (used as impostors
for acute infarction), and normal scans without calcification also were identified.
Scans with other abnormalities (including subarachnoid and extracerebral hemorrhage),
scans with multiple findings, and scans that could not be definitively placed
in 1 of these categories were excluded. Only scans for which there was unanimous
diagnostic agreement among the authors (and a consulting neuroradiologist)
were included. Using a consensus process, we classified each scan that demonstrated
hemorrhage as easy or difficult to interpret and each scan that demonstrated
acute infarction as easy, intermediate, or difficult to interpret based on
the subtlety of the findings.
Thirty-eight emergency physicians were recruited at the Scientific Assembly
of the California Chapter of the American College of Emergency Physicians
in May 1996. Seventeen community-based neurologists who periodically attend
at a university hospital neurology clinic were tested in private sessions
and 12 additional neurologists were tested while attending the University
of California at Los Angeles (UCLA) Stroke Center symposium, held in October
1996. Seven radiologists were contacted through the physician directors of
local community hospital radiology departments and tested at their hospitals.
Twenty-nine other radiologists were tested at the 49th Annual Midwinter Radiological/Oncology
Conference of the Los Angeles Radiological Society, held in January 1997.
Board-certified and board-eligible radiologists who did not have additional
training in neuroradiology were tested.
At the meetings, a booth was set up in the exhibits area of the conference
with a sign reading "Test Your Skills at CT." Subjects included physicians
who spontaneously approached the booth and those who, when approached, agreed
to participate in the study. At conference test sites, subjects were offered
a T-shirt as an incentive to participate and were informed that the physician
in each specialty with the highest score would receive a textbook of his or
her choice. Physician-subjects who gave verbal informed consent to participate
completed a single-sheet questionnaire regarding their age, years of clinical
experience, residency training, board certification, and typical involvement
in the reading of cranial CT scans. After completing the questionnaire, each
subject was shown the CT scans on a view box in an individual session with
unlimited time to interpret each scan. The study was approved by the UCLA
Institutional Review Board.
Physician-subjects were asked to assume that each scan was of a patient
who arrived at the hospital within the first few hours after the onset of
an acute hemispheric neurologic deficit (eg, aphasia, hemiparesis). As each
CT scan was presented, the physician-subject was told which side of the patient's
body was affected. Subjects were asked to accept that each patient was eligible
for thrombolytic therapy provided the CT scan had no contraindications. For
each patient, the physician was asked, "Based solely on scan findings, could
thrombolytics be administered to this patient?" Answer choices were (1) yes;
(2) no, because of hemorrhage; or (3) no, because of signs of acute infarction.
A list of contraindications (hemorrhage, early hypodensity, mass effect, and
shift) and a list of findings that did not preclude the administration of
thrombolytics (calcification, atrophy, and old infarction) were provided to
remind subjects of the criteria for this study. Subjects were informed that
20% to 60% of the scans would have no contraindication to thrombolytic therapy.
Each subject was presented with 5 initial scans: 2 difficult hemorrhages,
1 intermediate acute infarction, 1 impostor, and 1 normal (Figure 1). Subjects who responded correctly to all 5 scans were
placed in an advanced track and were then presented with 10 scans: 3 difficult
hemorrhages, 3 difficult acute infarctions, 1 intermediate acute infarction,
2 impostors, and 1 normal. Subjects who responded incorrectly to 1 or more
of the first 5 scans were placed in the standard track and presented with
3 difficult hemorrhages, 1 easy hemorrhage, 1 intermediate acute infarction,
2 easy acute infarctions, 1 impostor, and 2 normal scans. The 2-track strategy
was designed to ensure that subjects were given scans that were appropriate
to their skill levels and would maximize the discriminative capacity of the
test.13,14
Two hundred protocols, each containing a script of 25 scans, were created.
Each protocol specified that a subject must receive an initial series of 5
scans, with the scans to be included and their order of presentation determined
using the "uniform" random numbers function of STATA 5.0 (Stata Corp, College
Station, Tex). By a similar method, additional series of 10 scans each were
prepared for the standard and advanced tracks.
We designed the experiment to provide stable estimates of the performance
of each subject, the difficulty of each scan, and the sensitivity for detecting
hemorrhage, without unduly burdening the volunteers. Using a conservative
simulation, we determined that 30 subjects per specialty, each reading 15
scans, would meet these goals. Ninety-five percent confidence intervals (CIs)
surrounding percent correct values were calculated using robust clustered
logistic regression, which accounts for the fact that the scan readings may
be associated with the skills of the reader and, therefore, are not completely
independent.15
All 29 neurologists, all 36 radiologists, and 74% of the 38 emergency
physicians were board-certified (another 13% of the emergency physicians were
senior residents in emergency medicine). Emergency physicians averaged 9 years
in postresidency practice and 36 clinical hours per week, neurologists averaged
13 years in practice and 42 clinical hours per week, and radiologists averaged
15 years in practice and 42 clinical hours per week. Twenty-four percent of
emergency physicians routinely read cranial CT scans. All neurologists reported
reading CT scans; roughly half of them did so before seeing the radiologist's
report. All of the radiologists spent some time reading cranial CT scans (15%
of clinical practice on average), although 22% reported that the interpretation
of cranial CT and magnetic resonance imaging scans constituted less than 5%
of their practice.
There were few violations in protocol. One emergency physician was inadvertently
given only 14 scans. One radiologist was placed in the standard track despite
achieving a perfect score on the first 5 scans. Also, during testing of the
first group of radiologists it became clear that 2 scans, an intermediate
acute infarction and a difficult hemorrhage, were causing unintended confusion.
The difficult hemorrhage had an area that could have been read as infarction,
and subjects were retrospectively given credit for a correct answer provided
they did not score the scan as "give thrombolytics." As a result of this,
another radiologist was mistakenly placed in the standard track. These 2 cases
were removed and replaced with alternate scans that did not have these ambiguities.
Among all 103 physicians, 80 (78%) incorrectly interpreted at least
1 of the 5 scans in the initial series (Table 1). Five emergency physicians, 6 neurologists, and 10 radiologists
correctly responded to all 5 scans and entered the advanced track, achieving
average total scores of 67%, 80%, and 84%, respectively. Thus, even the best
performers misinterpreted a substantial number of scans.
Across all examinations, neurologists and radiologists were 100% accurate
for identifying easy hemorrhages. Emergency physicians identified 94% (95%
CI, 84%-98%) of these (Table 2).
For difficult hemorrhages, emergency physicians correctly read 56% (95% CI,
46%-67%), neurologists correctly read 78% (95% CI, 68%-85%), and radiologists
correctly read 80% (95% CI, 71%-87%), for an overall sensitivity for hemorrhage
of 82% (95% CI, 78%-85%). Fifty-two percent of radiologists, 40% of neurologists,
and 17% of emergency physicians correctly identified all CT scans with evidence
of hemorrhage.
Average correct score by all physicians on all examinations was 77%
(95% CI, 74%-80%) (Table 1). Overall,
of 569 CT scan readings by emergency physicians, 67% were correct; of 435
readings by neurologists, 83% were correct; and of 540 readings by radiologists,
83% were correct. Performance by neurologists and radiologists was weakly
inversely related to years in practice. Board certification in emergency medicine
did not predict higher performance. Emergency physicians who routinely read
the cranial CT scans they ordered scored higher than those who did not (71%
vs 59%; difference of 12%; 95% CI, 4%-21%).
Percent correct by scan type decreased sequentially from easy hemorrhages
(98% correct) through easy acute infarctions, normal scans, normal scans with
calcification, difficult hemorrhages, intermediate acute infarctions, normal
scans with old infarctions, and difficult acute infarctions (40% correct)
(Table 2). The correlation of
percent correct with scan difficulty (easy, intermediate, or difficult) substantiates
the validity of our classification schema.
Subjects in the advanced track (those who responded correctly to the
first 5 scans) consistently scored higher on each type of scan than those
in the standard track (Table 3).
For all but those in the lowest stratum of overall performance, easy hemorrhages
and easy acute infarctions were read with near perfect sensitivity. Scores
for other types of scans improved with increasing overall skill.
In the United States, recombinant tissue plasminogen activator is approved
for use in acute ischemic stroke when administered within 180 minutes of symptom
onset.1 During this time interval the patient
must recognize the symptoms, get to a hospital, undergo evaluation, and have
a CT scan performed and interpreted. There will be situations when the 3-hour
limit is rapidly approaching, no neuroradiologist is available, and an emergency
physician, neurologist, or general radiologist is the only physician available
to interpret the CT scan. Our study asks whether physicians in these specialties
are capable, without additional training, of interpreting the CT scan with
sufficient sensitivity to determine if thrombolytics may be administered safely.
Overall sensitivity for intracerebral hemorrhage, an absolute contraindication
to thrombolytic therapy, was 82%. Two levels of sensitivity for hemorrhage
may be expected of physicians who interpret cranial CT scans, one based on
a single-case perspective, the other based on a population perspective. From
the single-case perspective, the administration of a thrombolytic agent to
a patient with an intracerebral hemorrhage may be lethal or have other catastrophic
consequences. The potential consequences to the patient and the physician
of administering thrombolytic therapy after failing to recognize hemorrhage
on the CT scan mandate that sensitivity for intracerebral hemorrhage be extremely
high, certainly over 95% and ideally higher than 99%.
A population-based perspective provides a more relaxed requirement for
acceptable sensitivity. For instance, in the 624-patient National Institute
of Neurological Disorders and Stroke study, there were 10 more survivors in
the treatment group than in the control group at 3 months.4
Assuming that 15% of all acute hemispheric strokes are hemorrhagic,16 that patients with small hemorrhagic strokes have
prognoses equal to those with ischemic stroke, and that all patients with
hemorrhagic stroke who receive thrombolytics die, any sensitivity for hemorrhage
higher than 75% will preserve a potential net death benefit in the treated
group (unpublished data, D.L.S.). By similar reasoning, the sensitivity for
hemorrhage may decrease to as low as 22% before negating improvements in neurologic
outcome as measured with the Barthel Index.
We weigh heavily the single-case perspective and believe that, on average,
the sample of physicians in our study did not have the skills needed to recognize
hemorrhage on CT scans and determine which patients may safely receive thrombolytic
therapy. One implication of this finding is that physicians in these specialties
should not assume that the standard of care dictates that they should all
be able to make these decisions independently.
Sensitivity for identification of early signs of major infarction was
variable. Easy acute infarctions were identified by most physicians, but even
the best performers failed to identify a third of the difficult acute infarctions.
The clinical importance of this finding in patients with stroke within and
beyond 3 hours of symptom onset remains to be determined.10,17,18
Poor specificity would deprive eligible patients the opportunity to
benefit from thrombolytic therapy. While this is less catastrophic than administering
thrombolytic agents to patients with contraindications, our study demonstrates
that many physicians will have trouble differentiating hemorrhage from calcification
and acute infarction from old infarction.
It is possible that our study results could represent a biased estimate
of national average performance. Physicians in our study may not be representative
of the national population of physicians in each specialty, leading to sampling
bias. Furthermore, physicians taking a test may achieve results different
from those they achieve in actual practice. Motivation in the simulated testing
situation could be higher (eg, years of conditioning to perform well, desire
to win the textbook) or lower (eg, no patient's life is at stake) than in
clinical practice. We cannot predict the direction of this bias.
The inability of participants to view CT scans on a computer and measure
image density to differentiate calcium from blood may have adversely affected
specificity on normal scans with calcifications. Higher than ideal ambient
lighting in the testing area may have reduced overall performance. We intentionally
prevented physicians from stating "unsure" in response to our questions because
we were interested in determining what would happen if the physician were
the only person available to promptly read the scan. This approach most likely
decreased specificity (those physicians who expressed uncertainty about a
scan often scored it as infarction or hemorrhage because they did not wish
to miss contraindications), but should not have affected sensitivity.
The few breaks in protocol also could affect our estimates, but any
decrease in performance resulting from the 2 ambiguous scans was likely balanced
by the 2 subjects who were mistakenly placed in the standard track (where
they likely scored higher than they would have in the advanced track). Given
the magnitude of our results, and the fact that many physicians were able
to attain a perfect sensitivity for hemorrhage, we do not believe that these
testing biases are large enough to alter our findings substantially.
Our convenience sampling method is the main threat to the external validity
of the study. While board certification is an imperfect proxy for skill at
interpreting cranial CT scans, all neurologists and radiologists were board
certified, and the percentage of board-certified emergency physicians in the
sample exceeded the national percentage of full-time emergency physicians
who are board certified. While we cannot prove that our sample was representative
of the national population of physicians in these specialties, we observed
that physicians who acknowledged that they were uncomfortable reading CT scans
often declined participation, suggesting that our study most likely did not
underestimate physician performance. In addition, overall physician performance
correlated with performance on each scan type, which suggests that the examination
had construct validity.
In conclusion, it appears that while some members of each of these physician
groups are capable of identifying hemorrhage with perfect or near-perfect
sensitivity, the majority of those tested are not. Board certification in
emergency medicine, neurology, or general radiology is an inadequate marker
for such competence. Physicians involved in the care of patients with acute
stroke should ensure that the interpretion of the CT scan reliably identifies
intracranial hemorrhage when present. This may be accomplished by providing
physicians with enhanced training in the interpretation of cranial CT scans
or by implementing teleradiography or other systems that facilitate immediate
scan interpretation by qualified readers.
1.Nightingale SL. t-PA approved for acute ischemic stroke [From the Food and Drug Administration].
JAMA.1996;276:443.Google Scholar 2.Hacke W, Kaste M, Fieschi C.
et al. Intravenous thrombolysis with recombinant tissue plasminogen activator
for acute hemispheric stroke: the European Cooperative Acute Stroke Study
(ECASS).
JAMA.1995;274:1017-1025.Google Scholar 3.Multicenter Acute Stroke Trial–Italy (MAST-I) Group. Randomised controlled trial of streptokinase, aspirin, and combination
of both in treatment of acute ischaemic stroke.
Lancet.1995;346:1509-1514.Google Scholar 4.National Institute of Neurological Disorders and Stroke rt-PA Stroke
Study Group. Tissue plasminogen activator for acute ischemic stroke.
N Engl J Med.1995;333:1581-1587.Google Scholar 5.Donnan GA, Davis SM, Chambers BR.
et al. Streptokinase for acute ischemic stroke with relationship to time of
administration.
JAMA.1996;276:961-966.Google Scholar 6.Multicenter Acute Stroke Trial–Europe Study Group. Thrombolytic therapy with streptokinase in acute ischemic stroke.
N Engl J Med.1996;335:145-150.Google Scholar 7.Special Writing Committee of the Stroke Council, American Heart Association. Guidelines for thrombolytic therapy for acute stroke.
Stroke.1996;27:1711-1718.Google Scholar 8.Quality Standards Subcommittee of the American Academy of Neurology. Practice advisory: thrombolytic therapy for acute ischemic stroke:
summary statement.
Neurology.1996;47:835-839.Google Scholar 9. National Stroke Association consensus statement: stroke: the first
hours: emergency evaluation and treatment guidelines. In: Stroke Clinical Updates. Englewood, Colo: National
Stroke Association; 1997.
10.National Institute of Neurological Disorders and Stroke rt-PA Stroke
Study Group. Intracerebral hemorrhage after intravenous t-PA therapy for ischemic
stroke.
Stroke.1997;28:2109-2118.Google Scholar 11.Moulin T, Besson G, Crepin-Leblond T.
et al. Hemorrhagic transformations in MAST-E trial: predictive factors [abstract].
Cerebrovascular Dis.1996;6:182.Google Scholar 12.Larrue V, von Kummar R, del Zoppo G, Bluhmki E. Hemorrhagic transformation in acute ischemic stroke: potential contributing
factors in the European Cooperative Acute Stroke Study.
Stroke.1997;28:957-60.Google Scholar 13.Ebel RL, Frisbie DA. Essentials of Educational Measurement. 4th ed. Englewood Cliffs, NJ: Prentice-Hall International Inc; 1986:335-339.
14.Wainer H, Dorans NJ, Flangher R.
et al. Computerized Adaptive Testing: A Primer. Hillsdale, NJ: Lawrence Erlbaum Associates Inc Publishers; 1990.
15.Stata Corporation. Stata 5.0 Statistical Software User's Guide. College Station, Tex: Stata Corp; 1997:235-239.
16.Sacco R. Classification of stroke. In: Fisher M, ed. Clinical Atlas of Cerebrovascular Disorders. London, England: Mosby–Year Book Inc; 1994:2.1-2.23.
17.von Kummer R, Allen KL, Holle R.
et al. Acute stroke: usefulness of early CT findings before thrombolytic therapy.
Radiology.1997;205:327-333.Google Scholar 18.National Institute of Neurological Disorders and Stroke rt-PA Stroke
Study Group. Generalized efficacy t-PA for acute stroke: subgroup analysis of the
NINDS stroke trial.
Stroke.1997;28:2119-2125.Google Scholar