Electron-Beam Computed Tomography in the Diagnosis of Coronary Artery Disease: A Meta-analysis

Background  Electron-beam computed tomography (EBCT) is a new, noninvasive method of detecting coronary artery calcification that is being increasingly advocated as a diagnostic test for coronary artery disease (CAD). Before its clinical use is justified, however, the overall accuracy of EBCT must be better defined.

Objective  To estimate the accuracy of EBCT in diagnosing obstructive CAD.

Data Sources  English-language studies from January 1, 1979, through Feburary 29, 2000, were retrieved using MEDLINE and Current Contents databases, bibliographies, and expert consultation.

Study Selection  We included a study if it (1) used EBCT as a diagnostic test; (2) reported cases in absolute numbers of true-positive, false-positive, true-negative, and false-negative results; and (3) used coronary angiography as the reference standard for diagnosing obstructive CAD (defined as ≥50% diameter stenosis).

Data Extraction  Data were extracted from the included articles by 2 independent reviewers.

Data Synthesis  Weighted pooled analysis and summary receiver operating characteristic (ROC) curve analysis were used to determine sensitivity and specificity rates. Results from 9 studies with 1662 subjects were included. Pooled sensitivity for EBCT was 92.3% (95% confidence interval [CI], 90.7%-94.0%) and pooled specificity was 51.2% (95% CI, 47.5%-54.9%). Maximum joint sensitivity and specificity for EBCT from its summary ROC curve was 75%. As the threshold for defining an abnormal test varied, sensitivity and specificity changed. For a threshold that resulted in a sensitivity of 90%, specificity was 54%; when sensitivity was 80%, specificity rose to 71%.

Conclusion  The performance of EBCT as a diagnostic test for obstructive CAD is reasonable based on sensitivity and specificity rates from its summary ROC curve.

ELECTRON-BEAM computed tomography (EBCT) is a new, noninvasive method of obtaining cross-sectional images of the heart in subsecond scanning times.¹ Through the use of an electron x-ray source and 4 stationary tungsten targets, EBCT reduces the distorting effects of cardiac motion and yields high-resolution CT images with limitation of artifact.¹ Thus, EBCT can accurately detect and quantify even small areas of coronary artery calcification, an established marker for atherosclerosis and coronary artery disease (CAD).^1-4 As a noninvasive test for CAD, EBCT has great appeal because it is fast (<15 minutes for an entire scan), simple to use (no physiological or pharmacological intervention required), and predominantly operator independent.^5,6

During the next few years, the number of EBCT scanners in clinical use is anticipated to rise dramatically.⁵ Many of these scanners will be used to evaluate suspected obstructive CAD (defined as ≥50% diameter stenosis in any coronary vessel), one of the best-studied clinical indications for EBCT.^1,4 Support for its use in this setting has come from several sources. In a 1996 consensus statement, the American Heart Association declared EBCT to be "sufficiently accurate for predicting the presence of angiographic stenosis."⁴ Furthermore, evidence suggests that EBCT may be more cost-effective for diagnosing obstructive CAD than traditional noninvasive testing.⁷ Before clinical use of EBCT can be justified, however, its overall diagnostic accuracy, which has varied widely in published reports, must be more clearly defined.

We therefore performed a meta-analysis to answer the question "What is the overall discriminatory power of EBCT in diagnosing obstructive CAD in pa tients undergoing coronary angiography?" Although the reported sensitivity of EBCT ranges from 68% to 100% and specificity from 21% to 83%, some of the variability across different studies, as with all diagnostic tests, is likely due to differences in the test threshold used for defining an abnormal result.⁴ Until recently, no statistical framework existed to account for the contribution of these different "positivity" thresholds on the reported sensitivity and specificity of a test, which limited attempts to combine results quantitatively from different evaluations. However, with the development of summary receiver operating characteristic (ROC) curve analysis, threshold differences across studies can be adjusted for, thereby allowing for an objective review of available data.^8,9

We sought to summarize the discriminatory power for EBCT in diagnosing obstructive CAD and to determine if the performance of EBCT in diagnosing obstructive CAD was affected by selected study characteristics.

A computerized search was performed to identify relevant English-language articles published from January 1, 1979, through February 29, 2000, in MEDLINE and Current Contents databases. In MEDLINE, we combined medical subject headings tomography, x-ray computed or tomography scanners, x-ray computed with the exploded term coronary disease. We performed a similar Current Contents search crossing key words computed tomography or electron-beam computed tomography with coronary disease. We also scanned references in retrieved articles and contacted original authors and experts to identify other published or unpublished reports.

We included a study if it (1) used EBCT as a diagnostic test for obstructive CAD; (2) reported cases in absolute numbers of true-positive (TP), false-positive (FP), true-negative (TN), and false-negative (FN) results or presented sufficient data for deriving these numbers; and (3) used coronary angiography as the reference standard for diagnosing obstructive CAD. Studies were excluded if (1) performed exclusively in patients after coronary artery bypass graft surgery or percutaneous transluminal coronary angioplasty; (2) findings other than coronary artery calcification (eg, wall motion abnormalities and ventricular size) were used as criteria for a positive result; or (3) incomparable EBCT methods such as EBCT angiography or EBCT stress imaging were used. When several studies were published by a similar group of authors, the corresponding author from each report was contacted to determine whether significant overlap existed across the different study groups; if groups overlapped by more than 50%, we excluded results from the smaller study.

Two of us (B.K.N. and L.F.B.) independently extracted data from each article using a standardized form. Abstracted information included descriptive data (authors, title, journal citation, and year of publication), study group characteristics (sample size, mean age, proportion of men, and prevalence of obstructive CAD), study design characteristics (for EBCT this related to the threshold used to define a positive result and protocol information, whereas for the reference coronary angiogram this referred to the angiographic definition used for obstructive CAD), extent of blinding between readers of EBCT studies and the reference standard, and evidence of verification bias. For each study, results of the accuracy of EBCT were organized into a 2 × 2 table that classified outcomes as TP, FP, TN, and FN. When an article reported more than 1 set of results (using different positivity thresholds), the single set most emphasized by the authors was identified and tabulated for further analysis. Any inconsistencies or controversies encountered in abstracted data were resolved through discussion and consensus.

Pooled sensitivity and specificity estimates for EBCT were calculated using a fixed-effects model that weighted each report by its sample size.⁸ We constructed corresponding 95% confidence intervals (CIs) for sensitivity and specificity estimates.

We used a previously described method of variance-weighted least squares regression to estimate the summary ROC curve for EBCT.^9-12 Using data from the 2 × 2 table of each report, logit transformations of the TP rate (sensitivity) and FP rate (1 − specificity) were performed. The differences of the logit transformations (measures of the observed discriminatory power of EBCT) were then regressed on the sums of the logit transformations (measures of the positivity threshold used for determining a positive EBCT result). A summary ROC curve for EBCT was constructed by means of back transforming the fitted line from the regression model. We weighted each study in the regression model by its variance and restricted the final summary ROC curve to the range of observed TP and FP rates.

We characterized the summary ROC curve for EBCT by a point referred to as the "maximum joint sensitivity and specificity," or the upper-left-most point of the summary ROC curve.¹¹ This point is the maximum attainable common value of sensitivity and specificity and is an overall measure of the discriminatory power of a test (eg, a perfect test would have a maximum joint sensitivity and specificity of 100%). More important, this point does not indicate the only or even the best combination of sensitivity and specificity for a particular clinical setting. Rather, the summary ROC curve demonstrates the inherent trade-off that exists between sensitivity and specificity in any test as its positivity threshold is varied.

The addition of other covariates to the regression model allowed us to evaluate the impact of selected study characteristics on the overall discriminatory power of EBCT.^9-12 The β coefficients for covariates added to the model give an adjusted measure of test performance for selected characteristics, with positive coefficients indicating improved discriminatory ability and negative coefficients indicating less discriminatory ability. Selected characteristics that we examined as potentially related to test accuracy included sample size, mean age, proportion of men in the study group, percentage of study group with obstructive CAD, presence of blinding, and year of publication. The overall suitability of the pooled and summary ROC curve analyses was evaluated using the Spearman correlation coefficient.⁸ Heterogeneity was assessed in a standardized manner by determining whether the predicted discriminatory power for each study fell within the 95% CIs of the observed discriminatory power for each study.^9-12 All analyses were performed using commercially available software (Stata, version 5; Stata Corporation, College Station, Tex).

We retrieved 455 citations from the computerized search (431 citations were from MEDLINE; 38 were from Current Contents, and 14 were common to both databases). Reasons for exclusion are delineated in Table 1. Five studies were excluded because of a substantial degree of study group overlap.^13-17 Nine studies satisfied all inclusion criteria and were included in the meta-analysis.^18-26 One article, the multicenter study by Budoff and colleagues,¹⁸ reported results from 6 different sites (State University of New York–Buffalo; Harbor–University of California–Los Angeles Medical Center, Torrance; University of Illinois, Chicago; University of Iowa, Iowa City; Mount Sinai Hospital, Miami, Fla; and Washington State University, Spokane) as a single, pooled set of sensitivity and specificity rates.¹⁸ Because each center used different criteria in selecting a study population and interpreting EBCT results, the pooled set of results may have incorrectly estimated the overall accuracy of EBCT. We therefore obtained original data from the corresponding author (Matthew Budoff, MD) to calculate 6 separate center-specific TP and FP rates. A total of 14 reports from the 9 studies were included in our analysis.

Details from the included articles are summarized in Table 2. The mean number of subjects per report was 119, with study groups varying in size from 50 (Washington State University)¹⁸ to 251.²⁰ Reported sensitivity ranged from 81%¹⁹ to 99%²⁶ and specificity ranged from 14% (Washington State University)¹⁸ to 83%.²⁶ Most subjects were men (overall, 71%), with the proportion of men in each report ranging from 48%²¹ to 94%.¹⁹ Study groups in all instances consisted of individuals undergoing coronary angiography for evaluation of obstructive CAD. Five reports listed specific reasons for the referral.^19,22-25 Chest pain was the most common indication for coronary angiography in these studies, with the percentage of patients undergoing evaluation for this symptom ranging from 45%¹⁹ to 100%.^22,24 Other indications for coronary angiography included recent myocardial infarction, evaluation for valvular disease, and preoperative risk assessments. One study evaluated the accuracy of EBCT exclusively in patients with a decreased ejection fraction (<0.40) to assess its ability in distinguishing between nonischemic and ischemic causes of systolic dysfunction.²⁶

Similarities in EBCT protocols among the studies included the use of Agatston scoring, a standard definition of significant attenuation at 130 Hounsfield units, 100-millisecond scanning times, 3-mm slice thickness, and a scan acquisition trigger gated to 80% of the R-R interval. Variations between the study protocols existed and, most important, included different definitions of the minimum area of tissue attenuation required for a lesion to be considered calcium and not artifact; minimum areas ranged in size from 0.5^20,24,25 to 2 mm².²³ In general, study protocols were unclear as to the number of tomograms obtained in each examination (range, 20-40) and whether a standard inspiration or expiration breath-hold was used during image acquisition.

All studies except 1 assessed obstructive CAD similarly, defining it as at least a 50% diameter stenosis in any vessel.²⁰ Kajinami et al²⁰ used computer-assisted cinevideodensitometry to define significant angiographic obstruction as at least 75% densitometric narrowing. However, in cases where accurate densitometric measurements could not be made, a definition of obstructive CAD similar to the other studies (ie, ≥50% diameter stenosis) was used. Only 2 studies explicitly stated that blinding occurred between EBCT and the coronary angiogram readings.^20,26 Subjects in all studies underwent EBCT and coronary angiography, with no direct evidence that individuals were referred to angiography based on EBCT results. However, the possibility of verification bias could not be excluded definitively from any of the studies.

Pooled sensitivity for EBCT was 92.3% (95% CI, 90.7%-94.0%) and the pooled specificity was 51.2% (95% CI, 47.5%-54.9%). The summary ROC curve shown in Figure 1 summarizes joint TP and FP rates for EBCT from the included reports across a range of observed positivity thresholds. If a positivity threshold is used that yields a sensitivity of 80% for EBCT, specificity is likely to be 71%. If the positivity threshold is adjusted to increase sensitivity to 90%, specificity can be expected to fall to 54%. The maximum joint sensitivity and specificity rate for EBCT—the upper-left-most point in the summary ROC curve where sensitivity equals specificity—is 75%. This point is the maximum attainable common value of sensitivity and specificity for EBCT.

The only study characteristic found to have a significant impact on the discriminatory power of EBCT was sample size. Studies with more than 100 subjects reported higher discriminatory powers for EBCT than smaller studies (β coefficient, 0.81; 95% CI, 0.18-1.44; P = .01). The Spearman correlation coefficient between the TP and FP rates was 0.27 (P = .35), suggesting that it was appropriate to combine results using summary ROC analysis.⁸

Results from 4 reports demonstrated substantial heterogeneity (2 reports were from sites in the multicenter trial by Budoff and colleagues [State University of New York–Buffalo and University of Iowa]).^18,24,26 In 3 of the reports, no obvious differences in study design or patient characteristics could be identified, with predicted discriminatory powers for the report falling just outside the 95% CIs of the observed discriminatory powers for the report.^18,24 One study, however, examined the use of EBCT exclusively in patients with documented systolic dysfunction and reported a much greater discriminatory power for EBCT than predicted, which suggests that its accuracy in these individuals may be better than that in other groups.²⁶ More important, we found no substantial effect on the final results of the model after repeating our analysis with these 4 reports excluded. Our final results therefore include all 14 reports from the 9 studies.

We summarized the discriminatory performance of EBCT in diagnosing obstructive CAD through the use of a summary ROC curve, a new methodological tool for diagnostic meta-analyses.^8,9 Although a previous meta-analysis of EBCT accuracy in CAD has been published,²⁷ our evaluation was substantially different because we addressed the important issue of overlap between study groups and used summary ROC curve analysis to combine data. Summary ROC curves, like conventional ROC curves, graphically represent sets of sensitivity and specificity rates for a diagnostic test as the positivity threshold of a test is varied. However, although a conventional ROC curve summarizes the performance of a test in a single study population, a summary ROC curve describes a single set of operating characteristics for a test across multiple studies. Results from its summary ROC curve suggest that the diagnostic accuracy of EBCT in detecting obstructive CAD is reasonable with sensitivity and specificity rates comparable to those for traditional exercise stress testing.²⁸

Although the maximum joint sensitivity and specificity of EBCT from our analysis is near 75%, a single set of ideal sensitivity and specificity rates for EBCT cannot be determined from the summary ROC curve alone. The choice of an effective set of operating characteristics will need to be based on a clinician's desire for a particular specificity (increasing likelihood of false assumption of CAD with ≥50% angiographic stenosis) or sensitivity (assurance of not missing CAD with ≥50% angiographic stenosis). Clinicians, for instance, may use lower threshold criteria to increase the sensitivity of EBCT for detecting obstructive CAD in younger patients with atypical chest pain and no risk factors (ie, a low pretest probability of obstructive CAD). Conversely, higher positivity thresholds may be set when greater specificity is desired, as when examining older individuals with a higher likelihood of obstructive CAD. Unfortunately, our meta-analysis is unable to tell us how to set positivity thresholds explicitly to alter sensitivity and specificity rates in distinct patient populations or at different clinical centers. Although additional research is needed, the ease of altering positivity thresholds, through adjusting minimum size criteria for lesions or calcium scores,^4,16,17,23 is a potential advantage for EBCT compared with traditional noninvasive tests for CAD.

As a diagnostic test, EBCT has other desirable features. These include a brief and simplified testing protocol, a reasonable cost (estimated at $375-$450), and results that are reproducible and independent of the patient's effort or the test operator's experience.^1,4,6 Moreover, calcium scores from EBCT have been shown to correlate with the angiographic severity of CAD and may be useful in predicting left main or 3-vessel coronary disease.^16,18,29 Although evidence is still conflicting, EBCT results may also yield prognostic information about an individual's likelihood of future coronary events.^14,30,31 Finally, since hemodynamically significant lesions are required for a positive result in traditional noninvasive tests, EBCT may be the best and only noninvasive instrument that can accurately identify nonsignificant but clinically important CAD lesions. Although this may eventually become EBCT's most important advantage compared with traditional diagnostic tests for CAD, our meta-analysis was unable to address this issue because of a lack of current data on the accuracy of EBCT at detecting "subcritical" lesions in asymptomatic individuals.

There are several important limitations to the current use of EBCT that should be noted. First, there continues to be a wide variation in protocols for performing EBCT, which could lead to problems with its reproducibility and accuracy across facilities.²³ Prospective studies have also been unable to demonstrate that EBCT can localize "culprit" lesions. Furthermore, EBCT provides no physiologic or functional data and gives little or no information about left ventricular function or valvular disease.^1,4 Advanced imaging strategies, which include the use of intravenous contrast and/or stress images, are being developed to address several of these limitations.

In addition to summarizing the accuracy of EBCT, our study attempted to identify any study characteristics that could have influenced its performance. We found that larger studies (>100 subjects) reported small but significantly greater discriminatory powers, which suggests that a study group's size may influence the final outcomes. Although it has been suggested that the accuracy of EBCT may be improved in certain age and sex groups,¹⁵ we found no evidence to support such an association. Unfortunately, the small number of studies in this analysis limited our ability to detect such differences.

The results of our meta-analysis should be interpreted in context of the following limitations. First, studies included in our analysis varied widely in terms of study group demographics, prevalence of obstructive CAD, and EBCT protocol. Despite this variability, tests for heterogeneity did not indicate that the studies were significantly different except in 4 reports, and repeating our analysis after excluding these reports did not substantially affect our overall results. Second, all studies included in our analysis used study groups that consisted of individuals referred for coronary angiography. Given the current clinical indications for coronary angiography, it is likely that subjects from these studies were at high risk for obstructive CAD; thus, sensitivity and specificity rates could be different if this test was used in groups with a lower pretest probability of obstructive CAD. Third, several of the studies in our analysis had important limitations in study design that could have influenced their reported results. The lack of universal blinding between clinicians reading EBCT studies and the reference test, for instance, could have overestimated the accuracy of EBCT. Also, all studies were conducted in symptomatic patients who had their CAD status verified using coronary angiography findings and all studies were therefore subject to verification bias. This would have biased results toward an overestimation of sensitivity and an underestimation of specificity.³² Fourth, we estimated the sensitivity and specificity rates for EBCT at detecting at least a 50% diameter stenosis in any coronary vessel. We were not able to estimate its accuracy in lesions across a range of anatomic severity. Finally, publication bias is a possible weakness of any meta-analysis.^9,33 Although we attempted to locate unpublished studies, none were found. If studies with favorable results have a greater likelihood of being published, overall accuracy for EBCT may be inflated.

Our meta-analysis summarizes currently available data on the performance of EBCT in diagnosing obstructive CAD. Based on our results, EBCT appears to be reasonably accurate at detecting obstructive CAD in patients undergoing coronary angiography, with sensitivity and specificity rates comparable to those reported for traditional exercise stress testing. Of course, further studies are needed to clarify the accuracy of EBCT across atherosclerotic lesions of lesser or greater severity than those with a 50% diameter stenosis, at different explicit positivity thresholds, and to determine its exact role among the current armamentarium of noninvasive tests for CAD. Until then, however, our results provide clinicians with estimates of the overall accuracy of EBCT, allowing them to better interpret results from this rapidly diffusing diagnostic innovation.

Accepted for publication October 2, 2000.

Dr Nallamothu completed this work while under grant support from the Agency for Healthcare Research and Quality, Rockville, Md. This study was also supported in part by grant R01 HL-46292 from the National Institutes of Health, Bethesda, Md.

Presented in part at the 23rd Annual Meeting of the Society of General Internal Medicine, Boston, Mass, May 6, 2000.

We thank Matthew Budoff, MD, for providing original data for our analysis and for his review of the manuscript and helpful suggestions and Robert Detrano, MD, PhD, for his review of the manuscript and helpful suggestions.

Corresponding author and reprints: Brahmajee K. Nallamothu, MD, MPH, B1F245 University Hospital, Ann Arbor, MI 48109-0022 (e-mail: bnallamo@umich.edu).