Discordance Between Appropriate Use Criteria for Nuclear Myocardial Perfusion Imaging From Different Specialty Societies: A Potential Concern for Health Policy

Importance  Appropriate use criteria (AUC) assist health care professionals in making decisions about procedures and diagnostic testing. In some cases, multiple AUC exist for a single procedure or test. To date, the extent of agreement between multiple AUC has not been evaluated.

Objective  To measure discordance between the American College of Cardiology Foundation (ACCF) AUC and the American College of Radiology (ACR) Appropriateness Criteria for gauging the appropriateness of nuclear myocardial perfusion imaging.

Design, Setting, and Participants  Retrospective cohort study at an academically affiliated Veterans Affairs medical center. Participants were Veteran patients who underwent nuclear myocardial perfusion imaging between December 2010 and July 2011 with rating of appropriateness by the ACCF and ACR criteria. Analysis was performed in March 2015.

Main Outcomes and Measures  The primary outcome was the agreement of appropriateness category as measured by κ statistic. The secondary outcome was a comparison of nuclear myocardial perfusion imaging results and frequency of ischemia across appropriateness categories for the 2 rating methods.

Results  Of 67 indications in the ACCF AUC, 35 (52.2%) could not be matched to an ACR rating, 18 (26.9%) had the same appropriateness category, and 14 (20.9%) disagreed on appropriateness. The study cohort comprised 592 individuals. Their mean (SD) age was 62.6 (9.4) years, and 570 of 592 (96.2%) were male. When applied to the patient cohort, 111 patients (18.8%) could not be matched to an ACR rating, 349 patients (59.0%) had the same appropriateness category for the ACR and ACCF methods, and 132 patients (22.3%) were discordant. Overall, the agreement of appropriateness between the 2 methods was poor (κ = 0.34, P < .001). Ischemia was rare among patients rated as “inappropriate” by the ACCF AUC (1 of 39 patients [2.6%]), while ischemia was more common among patients rated as “usually not appropriate” by the ACR Appropriateness Criteria (14 of 80 patients [17.5%]).

Conclusions and Relevance  Substantial discordance may exist between methods for assessing the appropriateness of advanced imaging tests. Discordance in methods may translate into differences in clinically relevant outcomes, such as the detection of myocardial ischemia.

In 2014, the US Congress passed the Protecting Access to Medicare Act, directing the Secretary of Health and Human Services to use appropriate use criteria (AUC) in decisions for reimbursement of advanced imaging procedures, including nuclear myocardial perfusion imaging (MPI). Specific direction is provided that the AUC used should be “developed or endorsed by national professional medical specialty societies or other provider-led entities.”¹ For cardiac tests, such as nuclear MPI, multiple AUC have been developed, including those from the American College of Cardiology Foundation (ACCF) and the American College of Radiology (ACR).^2,3 The Centers for Medicare & Medicaid Services are in a rule-making process to implement their mandate. It remains unclear if they will permit multiple criteria for a single test or indication and how to resolve situations where multiple AUC exist. We applied the ACR Appropriateness Criteria² to a sample of patients previously rated by the ACCF AUC to identify differences and estimate the potential policy impact of multiple AUC.

Our group previously reported on a cohort of patients who underwent nuclear MPI between December 2010 and July 2011 at an academically affiliated Veterans Affairs medical center.⁴ The patients in this cohort were rated using the 2009 ACCF AUC, which were contemporary at the time.⁵ This investigation was reviewed by the University of Florida IRB-01 Gainesville Health Science Center Institutional Review Board, and the requirement for informed consent was waived.

We reviewed the ACCF AUC, attempting to match each indication to a clinical indication from the ACR Appropriateness Criteria. The ACCF AUC are structured such that precise clinical indications are individually rated. Each tends to incorporate multiple patient characteristics, such as symptoms, prior testing, cardiovascular risk, and the motive for testing (ie, preoperative evaluation). In contrast, the ACR criteria are broader categories with fewer such distinctions, usually based only on symptoms and risk. Examples include asymptomatic patients with low risk of coronary artery disease or chronic chest pain with high risk of coronary artery disease. In areas of agreement, a match was made. In some cases, a risk assessment had to be estimated from an ACCF indication to match with an ACR rating. For example, the ACCF AUC indication 55 (“evaluation of ischemic equivalent” in a patient after revascularization) was matched to the ACR rating “chronic chest pain with high risk of coronary artery disease.” The ACCF AUC provide ratings for specific cardiac indications, such as before cardiac rehabilitation, ventricular tachycardia, or elevated troponin without evidence of acute coronary syndrome, while these indications are absent from the ACR criteria. In these cases, a match was not made between the ACCF and ACR. The existing ACCF ratings were used to impute an ACR rating for each patient through the ratings that were matched. If a patient underwent an ACCF-rated indication for which no ACCF to ACR indication match had been made, then no ACR rating was imputed (eTable in the Supplement).

We compared categorical appropriateness ratings between the ACCF and ACR methods by κ statistic. The ACCF method uses the terms “inappropriate,” “uncertain,” and “appropriate,” while the ACR method uses the terminology “usually not appropriate,” “may be appropriate,” and “usually appropriate” to describe categories of appropriateness. We compared nuclear MPI results and the prevalence of ischemia (defined as a summed difference score >3) within each appropriateness method by χ² test. Statistical significance was predefined as P < .05.

The 2009 ACCF AUC include 67 indications, of which 32 could be successfully paired to 1 of 7 indications from the ACR. For 18 indications, the appropriateness rating did not change, and the remainder of pairings disagreed on appropriateness (Figure). The 2 rating methods were applied to a predominantly male (96.2% [n = 570]) cohort of 592 patients. Obesity was common (67.5% [n = 397]), as were hypertension (82.5% [n = 487]), diabetes mellitus (41.4% [n = 244]), prior coronary artery disease (41.2% [n = 243]), and current smoking (25.8% [n = 152]). For 111 patients (18.8%), no matched rating could be made from the ACCF to the ACR criteria. Of the successfully paired patients, 72.6% (349 of 481) had agreement on the appropriateness of nuclear MPI (κ = 0.34, P < .001) (Table 1).

Among nuclear MPI rated as “inappropriate” by the ACCF criteria, few were abnormal (3 of 41 patients [7.3%]) or demonstrated ischemia (1 of 39 patients [2.6%]). “Appropriate” and “uncertain” nuclear MPIs were significantly more likely to be abnormal (P = .002) or ischemic (P = .03) (Table 2 and Table 3). When rated by the ACR criteria, “usually not appropriate” tests were frequently abnormal (29 of 82 patients [35.4%]) or showed ischemia (14 of 80 patients [17.5%]). Myocardial perfusion imaging rated as “may be appropriate” by the ACR criteria was significantly less likely to be abnormal (P = .004), and the presence of ischemia was not significantly different from the other categories (P = .07).

Only 2 rating matches resulted in a disagreement in which nuclear MPI was rated as “appropriate” by one method and “inappropriate” by the other. The ACCF AUC indication 1 (“nonacute ischemic equivalent with low pretest probability of coronary artery disease, able to exercise, and normal [electrocardiogram]”) is rated as “inappropriate” and was matched to the ACR indication “chronic chest pain with low to intermediate probability of coronary artery disease,” which rated nuclear MPI as a score of 8 (“usually appropriate.” In the second disagreement, the ACCF AUC indication 58 (“asymptomatic ≥5 years after bypass grafting surgery”) is rated as “appropriate” and was matched to the ACR criterion “asymptomatic patient at intermediate risk for coronary artery disease,” with a score of 2 (“usually not appropriate”). These 2 disagreements accounted for 9 and 10 patients, respectively.

In this pilot attempt to compare appropriateness ratings from methods developed by different specialty societies, we observed substantial discordance when appropriateness ratings were compared at the patient level. Most ACCF ratings could not be paired to an ACR rating, which limits our ability to assess agreement on appropriateness ratings but, most important, speaks to the limited interoperability of these 2 methods. For our investigation, we elected to match the 2 rating methods at the level of the indications described. If repeated by rating each patient based on his or her individual clinical presentation, the results may be different. While most ratings were unmatched, when measured as a proportion of patients, only 18.8% (111 of 592) did not have matched ratings, suggesting that the areas of discordance may be smaller.

Most disagreements in ratings were just one category apart, while only 2 indications had strong disagreement (ie, “appropriate” in one and “inappropriate” in the other). As new iterations of ratings are published, disagreements between the 2 rating methods may be reduced. One of the substantial areas of disagreement was for high-risk asymptomatic patients. In the 2009 ACCF AUC used in this investigation, this nuclear MPI was rated as “appropriate,” whereas the ACR rated it as “may be appropriate.” With publication of the new ACCF multimodality AUC³ in 2013, nuclear MPI for high-risk asymptomatic patients now is rated as “may be appropriate,” in agreement with the ACR. In another instance of disagreement, the 2009 ACCF AUC rated nuclear MPI for asymptomatic patients at least 5 years after bypass surgery as “appropriate.” In the ACCF multimodality AUC published in 2013, nuclear MPI was rated as “may be appropriate,” reducing disagreement with the ACR method. In the cardiology literature on appropriateness, many studies have reported on subsequent outcomes. While data on hard end points, such as death and infarction, are limited, nuclear MPI rated as “inappropriate” by the ACCF has been consistently reported as less likely to be abnormal and demonstrating less ischemia than “uncertain” or “appropriate” nuclear MPI.⁶ Fewer data are available from the literature on the ACR criteria. Given the exploratory nature of our investigation, we would not conclude that the ACR criteria are less accurate in this regard but would encourage researchers to report more data applying the ACR criteria so that sound judgments can be made about their clinical usefulness and accuracy.

Our study has some limitations. This investigation was conducted using readily available data from the Malcom Randall Veterans Affairs Medical Center, in which appropriateness was rated with the 2009 ACCF AUC, which is a major limitation given that these criteria have been supplanted by the updated 2013 ACCF multimodality AUC. Further study in a more contemporary cohort and using the newest AUC is warranted. The demographics of the Veteran population studied in this investigation are unique, and the results may differ in other populations. Our methods are but one way to compare these 2 appropriateness rating methods, and we would encourage others to investigate alternative strategies to assess them.

Some advanced imaging tests have multiple sets of AUC. Owing to differences between AUC, the appropriateness of testing for some indications may be discordant. The capacity for AUC methods to identify clinically relevant outcomes, such as the detection of myocardial ischemia, warrants further study.

Accepted for Publication: January 7, 2016.

Corresponding Author: David E. Winchester, MD, MS, Medical Service, Malcom Randall Veterans Affairs Medical Center, 1601 SW Archer Rd, PO Box 111-D, Gainesville, FL 32608 (david.winchester@va.gov).

Published Online: March 23, 2016. doi:10.1001/jamacardio.2016.0030.

Author Contributions: Dr Winchester had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Winchester.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Winchester, Wolinsky.

Critical revision of the manuscript for important intellectual content: Winchester, Beyth, Shaw.

Statistical analysis: Winchester, Beyth.

Administrative, technical, or material support: Beyth.

Study supervision: Winchester, Shaw.

Conflict of Interest Disclosures: Dr Winchester reported being a member of the American College of Cardiology Foundation Appropriate Use Criteria Task Force, which did not have any relationship with this investigation. No other disclosures were reported.

Funding/Support: This investigation was supported by T35 Training Grant T35-HL007489-28 from the National Institutes of Health. This material is the result of work supported with the resources of and the use of facilities at the Malcom Randall Veterans Affairs Medical Center.

Role of the Funder/Sponsor: The sponsors had no input on the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Disclaimer: The contents do not represent the views of the US Department of Veterans Affairs or the United States Government.

Previous Presentation: These data were presented at the 20th Annual Scientific Session of the American Society of Nuclear Cardiology; September 19, 2015; Washington, DC.

Additional Contributions: Ryan Meral, MD (anesthesiology resident at the University of Michigan, Ann Arbor), who was compensated under the T35 Training Grant, contributed to the collection of data for this investigation.