Attending academic cardiologists were randomized into control and intervention groups. Control and intervention groups were similar in terms of composition by academic cardiologists’ areas of subspecialty practice and academic rank.
aP = .86 for difference by practice area between intervention and control groups.
bP = .99 for difference by academic rank between intervention and control groups.
Month-by-month data and overall mean showing the percentages of rarely appropriate TTEs ordered during the study period in the intervention and control groups.
Individualized academic cardiologist data show the aggregate percentage of appropriate transthoracic echocardiograms (TTEs) and rarely appropriate TTEs ordered during the study by each cardiologist in the control group (A) and intervention group (B). In the control group, cardiologist 1 had neither appropriate nor rarely appropriate TTEs and so has 0% on the bar graph. In the intervention group, cardiologist 1 did not order any TTEs during the study period and so has 0% on the bar graph.
eTable 1. Reasons for Unclassifiable TTEs
eTable 2. Common Appropriate and Rarely Appropriate Indications for TTEs
eFigure. Rates of Rarely Appropriate TTE Ordering Stratified by Academic Rank
Customize your JAMA Network experience by selecting one or more topics from the list below.
Dudzinski DM, Bhatia RS, Mi MY, Isselbacher EM, Picard MH, Weiner RB. Effect of Educational Intervention on the Rate of Rarely Appropriate Outpatient Echocardiograms Ordered by Attending Academic Cardiologists: A Randomized Clinical Trial. JAMA Cardiol. 2016;1(7):805–812. doi:10.1001/jamacardio.2016.2232
Does an educational and feedback intervention reduce rarely appropriate transthoracic echocardiograms (TTEs) ordered by attending academic cardiologists?
In this randomized clinical trial of 65 academic cardiologists, the proportion of rarely appropriate TTEs (judged by published appropriate use criteria for echocardiography) was significantly lower in the intervention group (10.5%) compared with the control group (16.5%).
Targeted educational interventions for attending cardiologists may improve the utilization of echocardiography by reducing rarely appropriate studies.
Appropriate use criteria–based educational initiatives have been shown to improve transthoracic echocardiography (TTE) ordering practices of physicians in training. Whether such an intervention is successful with attending cardiologists remains unknown.
To prospectively investigate the effect of an appropriate use criteria–based educational intervention on ordering of outpatient TTEs by attending academic cardiologists.
Design, Setting, and Participants
We conducted a prospective, randomized clinical trial of an educational intervention designed to reduce the number of outpatient TTEs that were deemed to be rarely appropriate by published appropriate use criteria. Investigators classifying TTEs were blinded to participant groupings. The study was conducted within the cardiology division at the Massachusetts General Hospital, an academic quaternary care hospital. Staff members of the cardiology division were included; 66 cardiologists were randomized. The study was conducted from November 19, 2013, to June 1, 2014. An analysis of the evaluable population was performed.
The appropriate use criteria–based educational intervention consisted of a review lecture and electronic information card, as well as monthly individual physician feedback via email. The email described the percentage of rarely appropriate TTEs as well as the appropriate use criteria rationale for classifying studies as rarely appropriate.
Main Outcomes and Measures
We hypothesized a priori that the educational intervention would reduce the number of rarely appropriate TTEs. The primary outcome was the rate of rarely appropriate TTEs.
Of the 66 cardiologists enrolled in the study, 65 were included in the analysis (1 intervention cardiologist retired from practice during the study). The participants’ mean (SD) age was 50.6 (10.5) years; 48 (73%) were men. Following intervention, the proportion of rarely appropriate TTEs was significantly lower in the intervention vs control group (143 of 1359 [10.5%] vs 285 of 1728 [16.5%]; odds ratio [OR], 0.59 [95% CI, 0.39-0.88]; P = .01), and there was a nonsignificant increase in the proportion of appropriate TTEs in the intervention vs control group (1054 [77.6%] vs 1244 [72.0%]; OR, 1.38 [95% CI, 0.93-2.05]; P = .11). The most common of the 428 rarely appropriate indications were routine surveillance within 3 years after prosthetic valve insertion (73 [17.1%]), routine surveillance within 1 year for moderate or severe valvular stenosis (64 [15.0%]), and routine surveillance of cardiomyopathy (45 [10.5%]) or ventricular function (36 [8.4%]).
Conclusions and Relevance
An appropriate use criteria–based educational and feedback intervention reduced the number of rarely appropriate TTEs ordered by attending academic cardiologists. This strategy may be feasible to improve TTE utilization among cardiologists, and this type of intervention warrants study in other practice environments.
clinicalrials.gov Identifier: NCT01968642
Appropriate use criteria (AUC) were developed out of concern regarding increased use of noninvasive cardiac imaging services and Medicare spending between 1995 and 2006.1 The American College of Cardiology Foundation published its first AUC document in 2005.2 The AUC describe consensus ratings of the suitability of an imaging test or intervention in a particular clinical scenario, thus guiding physicians in decision making. Appropriate use criteria presently exist for diagnostic imaging and procedures in cardiology3,4 as well as in other specialties.5,6 Despite AUC being available for more than a decade, there is still need for improvement in utilization of cardiovascular testing.7
Appropriate use criteria represent a possible method to regulate growth of imaging services and health care costs. Few studies have assessed the ability of AUC-based educational interventions to change physicians’ behavior. Such studies8-12 have evaluated the effect of AUC on echocardiography, single-photon emission computed tomography, and coronary computed tomographic angiography, achieving various levels of success. Studies of transthoracic echocardiography (TTE) have been limited by the lack of randomized clinical trial design8 or inclusion of only physicians in training.9
Whether an AUC-based educational intervention can reduce rarely appropriate TTEs ordered by attending-level academic cardiologists is unknown. It is possible that physicians in training may be more responsive to feedback-based interventions, thereby resulting in success of the previous study.9 We therefore designed and conducted what we believe to be the first randomized clinical trial of an AUC-based educational and feedback intervention to determine whether such an intervention would reduce the rate of rarely appropriate outpatient TTEs ordered by attending academic cardiologists.
We conducted a prospective, randomized clinical trial of an AUC-based educational and feedback intervention designed to reduce rarely appropriate outpatient TTEs ordered by attending academic cardiologists. We randomized cardiologists 1:1 by random number generator into control and intervention groups (Figure 1). The study period was November 19, 2013, to June 1, 2014. The study protocol (Supplement 1) was reviewed and approved by the Partners HealthCare Institutional Review Board. Participant physicians provided verbal consent prior to study initiation; no financial compensation was provided.
This study was conducted in ambulatory cardiology practices at Massachusetts General Hospital, a quaternary care academic medical institution in Boston. Study participants were cardiologists on staff at Massachusetts General Hospital with a hospital clinical appointment and an academic appointment at Harvard Medical School, as well as an active outpatient cardiology practice. Participant academic cardiologists practice in noninvasive, heart failure and transplant, electrophysiology, or interventional cardiology settings. These staff cardiologists care for a variety of patients, including new outpatient consults, established patients, and posthospitalization follow-up patients. Each staff cardiologist has patients with general cardiovascular problems, but most practice within an area of special expertise. A TTE is typically not preordered for initial or follow-up visits unless the cardiologist has made a decision to order the study. All orders for TTEs were made by the cardiologist, and TTEs ordered by other health care professionals working with a cardiologist (ie, nurse practitioners) were excluded. During the study period no decision-support tool existed at the point of TTE ordering.
Physicians in the intervention group received a lecture on AUC concepts and a review of common clinical scenarios in which TTE is appropriate and rarely appropriate. After each calendar month, physicians in the intervention group received individualized email feedback documenting the total numbers of TTEs ordered and how many were classified as appropriate, may be appropriate, and rarely appropriate by 2011 AUC.3 This terminology was consistent with the AUC methodology and nomenclature update.13 For TTEs characterized as rarely appropriate, an explanation of the rationale for this classification was provided in the feedback.
Physicians in the control group received the same lecture and had TTE ordering tracked and classified but received no feedback on ordering behavior. Study participants could not be blinded to their study arm but were blinded to the assignment of their colleagues. Participants were instructed to not discuss the study or group assignment with peers, although this could be neither strictly monitored nor enforced. Investigators conducting TTE classification were blinded to the assignments of a cardiologist to the intervention or control group.
Lists of TTEs ordered during the study period by cardiologists in the control and intervention groups were created; TTEs performed for research were excluded. The date and clinical rationale for each TTE were ascertained by thorough review of the electronic medical record (EMR).9 At our institution, the EMR is a comprehensive database composed of inpatient and outpatient data, including notes, laboratory tests, and diagnostic studies. The EMR also includes records from 10 other hospitals in the Partners HealthCare network; TTE reports from other facilities are frequently scanned into the EMR.
Individual TTEs were classified by study investigators (D.M.D. and R.B.W.) according to 2011 AUC after a detailed review of EMRs with reference to the cardiologist’s stated indication, purpose, and clinical concerns as well as patients’ symptoms, signs, and results from prior testing. If the investigators disagreed on categorization and were unable to reach a consensus, a third study investigator (M.H.P.) was available to adjudicate. All TTEs were classified as appropriate, may be appropriate, or rarely appropriate; if the TTE could not be fit into one of the existing 2011 AUC indications, it was considered unclassifiable. Investigators were blinded to cardiologist assignment to the control or intervention group so as to not bias classifications. Outside TTE reports and images were used to assist in classification if sufficient information was available or, conversely, if the cardiologist’s note indicated that the outside study was incomplete or unable to answer the clinical question. Patient demographics and comorbidities were determined through automated EMR review and classified according to International Classification of Diseases, Ninth Revision, codes.
The primary outcome measure was the rate of rarely appropriate TTEs. Secondary outcomes included rates of appropriate TTEs and may be appropriate TTEs. The rate of unclassifiable TTEs and the most common appropriate and rarely appropriate indications for TTEs were also assessed.
The baseline rate of rarely appropriate TTEs was 22.0% according to previous retrospective data from our institution.14 Using logistic regression analysis with PASS, version 12 software (NCSS Statistical Software, LLC), we calculated that a total of 440 TTEs, split evenly between intervention and control groups, was required to detect a 10% absolute reduction in the rarely appropriate ordering rate, with 80% power and at significance level of α = .05. Categorical variables for cardiologists were compared using the Fisher exact test; continuous variables were compared using an unpaired, 2-tailed t test. The median numbers of TTEs ordered by cardiologists in each group were compared using the Wilcoxon rank sum test. We compared the absolute rates of rarely appropriate, may be appropriate, and appropriate TTEs ordered in the intervention and control groups using mixed-effects logistic regression incorporating cardiologists as random effects, with random intercepts to account for any intercardiologist differences in ordering.15 Unclassifiable TTEs were excluded from analyses. Statistical analyses were carried out using RW, version 3.2.2 (R Foundation; https://www.r-project.org/). Statistical significance is indicated by P < .05.
A total of 66 staff cardiologists participated, with 33 cardiologists randomized to the control group and 33 randomized to the intervention group. The participants’ mean (SD) age was 50.6 (10.5) years; 48 (73%) were men. One cardiologist in the intervention group retired after randomization and was excluded from analysis. Practice subspecialty and academic rank were similar between the control and intervention groups (Figure 1).
From November 19, 2013, to June 1, 2014, there were a total of 19 691 patient visits to staff cardiologist outpatient practices: 9750 in the control group and 9941 in the intervention group, with 7909 and 8116 unique patients, respectively. Table 1 compares the patient characteristics between the 2 groups. There were many statistically, but not clinically, significant differences present, with most absolute differences (except hyperlipidemia) less than 3%.
During the study, a total of 3193 TTEs were ordered: 1775 by cardiologists in the control group (55.6%) and 1418 by cardiologists in the intervention group (44.4%). There was a skewed distribution of TTEs ordered and a nonsignificant difference in the median number of TTEs ordered by physicians in the control group (30 TTEs per cardiologist; mean [SD], 53.8 [54.6]) vs the intervention group (36 TTEs per cardiologist; 44.3 [33.4]) (P = .90).
Using the 2011 AUC, 308 of 3193 TTEs (96.7%) were classifiable. The most common reasons for unclassifiable TTEs most often involved atrial fibrillation (follow-up or before ablation) (eTable 1 in Supplement 2). Of classifiable TTEs, the proportion of those that were rarely appropriate was significantly lower in the intervention group (143 [10.5%] of 1359) than in the control group (285 [16.5%] of 1728) (odds ratio [OR], 0.59 [95% CI, 0.39-0.88]; P = .01) (Table 2). The proportion of appropriate TTEs in the intervention group (1054 [77.6%] of 1359) was higher than that in the control group (1244 [72.0%] of 1728), but the difference was not significant (OR, 1.38 [95% CI, 0.93-2.05]; P = .11). Month-by-month data for rarely appropriate TTEs in the intervention and control groups are shown in Figure 2. When the date on which a TTE was performed was added to the regression model, both the date and the intervention were significant for a reduction in the rate of rarely appropriate TTEs (P = .01), indicating that, although the overall rate of rarely appropriate TTEs decreased over time, the intervention group demonstrated a significantly greater reduction over time than the control group.
Analysis of ordering patterns by individual staff cardiologists showed that 25 of the 32 participants (78.1%) in the intervention group ordered at least 1 rarely appropriate TTE; 1 of 32 physicians (0.31%) in the intervention group ordered no TTEs during the study period. Similarly, 29 participants (87.9%) in the control group ordered at least 1 rarely appropriate TTE. The individual ordering patterns of staff cardiologists are shown in Figure 3; percentages for any cardiologist may not total 100% if there were TTEs classified as may be appropriate.
When stratified by academic rank (eFigure in Supplement 2), all ranks demonstrated a trend to a lower (or equivalent) rate of rarely appropriate TTEs in the intervention vs control group. Adding academic rank to the mixed regression model showed that no group had a significantly higher rate of rarely appropriate TTEs.
The most common appropriate and rarely appropriate indications for TTEs ordered during the study are reported in eTable 2 in Supplement 2. All of the most common rarely appropriate indications were AUC scenarios classified as routine surveillance, which excludes situations in which there was a change in symptoms, clinical status, examination, or other findings.3 Specific rarely appropriate indications included surveillance within 3 years of prosthetic valve implantation (73 [17.1%] of 428), surveillance within 1 year for moderate or severe valve stenosis (64 [15.0%] of 428), surveillance within 1 year for known cardiomyopathy (45 [10.5%] of 428), and surveillance of ventricular function (36 [8.4%] of 428).
To our knowledge, this study was the first prospective, randomized clinical trial of an AUC-based educational intervention aimed at outpatient TTE ordering by attending academic cardiologists. The educational and feedback intervention reduced the rate of rarely appropriate TTEs ordered by staff cardiologists at an academic medical center. Scheduled provider-level feedback may therefore represent a viable strategy to improve practice patterns and optimize utilization of outpatient TTE. Because more than half of rarely appropriate TTEs were performed for routine surveillance of cardiac conditions in the absence of clinical changes, focusing educational efforts on such scenarios may improve adherence to AUC-based practices.
Appropriate use criteria were developed in response to increasing use of noninvasive cardiac imaging services and resultant health care costs. For echocardiography, initial AUC for TTE were published16 in 2007 and updated3 in 2011. Appropriate use criteria have allowed for characterization of practice patterns and determination of appropriateness rates. One common finding has been a higher rate of rarely appropriate TTEs in the outpatient vs inpatient setting.14 “Surveillance” studies, referring to several TTEs in patients with known cardiovascular disease but neither changes in clinical status nor physical examination findings, typically drive higher outpatient rates of rarely appropriate TTE,14,17 as was the case in the present investigation.
Educational intervention studies have been designed to actively use AUC to educate ordering clinicians in efforts to reduce the number of rarely appropriate imaging studies. The literature base in this field is relatively small, and studies to date18 have achieved mixed results. Attempts to apply AUC to improve utilization of single-photon emission computed tomography and stress echocardiography met with limited success.10,11 In contrast, in a study of coronary computed tomographic angiography, rarely appropriate studies decreased 60% (intervention included educational conferences and possible loss of third-party payer coverage).12 For TTEs, the first such AUC-based educational study, conducted on inpatient medical services, showed that a didactic lecture, pocket card applying the AUC for common clinical scenarios, and twice-monthly feedback emails resulted in a significant reduction of rarely appropriate TTEs.8 A subsequent randomized study9 of an AUC educational and feedback intervention aimed at physicians in training in an outpatient cardiology environment resulted in the proportion of rarely appropriate TTEs being significantly lower in the intervention vs control group.
The present investigation adds to the literature base in this emerging field as what we believe to be the first study of an AUC-based educational intervention aimed at TTE ordering by attending academic cardiologists. It could be hypothesized that these cardiologists would be less likely than trainees to change their ordering practice based on education and feedback; however, our study suggests that attending academic cardiologists may be responsive and therefore a target population for further programs aimed at improving practice. In addition, the present study is, to our knowledge, only the second randomized clinical trial of an AUC-based educational and feedback intervention, with the first such trial being limited to physicians in training.9 Our findings are consistent with prior results10,18 that more intensive intervention involving active engagement and feedback has a greater likelihood to effect change than relatively passive interventions.
Results from our study indicate that attending academic cardiologists can amend their ordering of outpatient TTEs in response to education and feedback. Within each academic stratum, cardiologists in the control group ordered more rarely appropriate TTEs compared with cardiologists in the intervention group. This finding suggests that the educational intervention used in the present study can be successful with most academic cardiologists along a spectrum of experience levels, although the most accomplished cardiologists (professors) may have more fixed behavior. Conversely, we could have hypothesized that junior faculty—trained in an era of cost-consciousness—may exhibit better adherence to AUC. Identifying groups that are most in need of and responsive to AUC interventions represents important future work.
Our study indicated that a relatively small number of clinical indications make up the majority of rarely appropriate TTEs in the outpatient setting. These TTEs focus on reassessment of patients with known cardiovascular disease (eg, valvular or heart muscle disease) but without changes in clinical status or physical examination. In this setting, TTEs ordered before a prespecified time interval (ie, within 3 years in the case of mild valvular regurgitation and/or stenosis) are classified as rarely appropriate. This finding supports prior retrospective data using AUC14 and indicates that tailoring educational interventions to these highest-yield indications may be necessary to produce measurable improvements in ordering behavior. In addition, this study provides novel data regarding unclassifiable TTEs. Although the 2011 AUC significantly reduced the number of unclassifiable TTEs,14,17 a small percentage remain unclassifiable. The present study, to our knowledge, represents the largest AUC study of TTE ordering (>3000 TTEs), and several indications emerged as the most common reasons for unclassifiable TTEs. These included atrial fibrillation (follow-up or preablation TTEs), serial cardiac resynchronization therapy follow-up without worsening heart failure or suspicion of device dysfunction, and left ventricular function assessment after nonacute coronary syndrome revascularization. It is possible that the need for practice improvement in these domains is greater than believed, since the AUC do not readily capture clinical practice in these particular settings.
There are several limitations of this study. First, the trial was aimed at attending academic cardiologists, and the effect of performing this type of intervention on attending physicians from other disciplines (ie, general internists) is unknown. Noncardiologists (eg, primary care, family practice, surgeons, neurologists) ordered up to half of the TTEs in a large Medicare database19; therefore, any systemic efforts will need to include physicians other than cardiologists. Similarly, studies need to involve nurse practitioners who may work closely with cardiologists and order TTEs. Because the study was limited to attending academic cardiologists, the number of participants was relatively small, with only 33 physicians assigned to each group. Second, this study was performed at an academic center, where several specialized referral cardiac programs (eg, interventional valvular disease, thoracic aortic disease, and adult congenital heart disease) exist; therefore, our findings may not be generalizable to other practice environments. Such limitations will be addressed by an ongoing, multisite international trial of an AUC-based educational intervention aimed at improving appropriate use of TTEs in various settings.20 In the present study, there was a nonsignificantly increased number of TTEs ordered in the control group, and it is unclear whether this greater number was attributable to differences in the patient population (ie, more myocardial infarctions, cancer, and atrial fibrillation in the control group) or baseline physician preferences and ordering practices. In addition, the control group’s rarely appropriate rate was lower than anticipated at 16.5% vs 22.0%, but this rate may reflect better knowledge possessed by attending academic cardiologists regarding appropriate utilization of TTE. Furthermore, the rarely appropriate TTE rate was lower in the intervention than control group in the first month of the study, although this difference was not statistically significant (P = .64). Both groups received an introductory lecture, but perhaps knowing that they were to receive individual feedback stimulated a change in ordering from the outset in the intervention group. The possibility that the observed changes in appropriate and rarely appropriate rates result from changes in documentation, rather than actual changes in practice, cannot be excluded. Moreover, academic physicians may be more motivated to alter practice patterns in the context of a research protocol. The sustainability of the impact of this type of intervention needs further study since there are discrepant data on the long-lasting effects of these types of interventions.12,21 Finally, the effect of adherence to AUC on patient outcomes and cost-effectiveness requires future study.
We report what we believe to be the first prospective, randomized clinical trial of an AUC-based educational intervention aimed at ordering of outpatient TTEs by attending academic cardiologists. The educational and feedback intervention reduced ordering of rarely appropriate TTEs, which indicates that this type of intervention may be used to improve TTE utilization. Educational and feedback interventions warrant study in various practice environments and with other types of ordering health care professionals.
Corresponding Author: Rory B. Weiner, MD, Cardiac Ultrasound Laboratory, Cardiology Division, Massachusetts General Hospital, 55 Fruit St, Yawkey Ste 5B, Boston, MA 02114 (email@example.com).
Accepted for Publication: June 1, 2016.
Published Online: August 10, 2016. doi:10.1001/jamacardio.2016.2232.
Author Contributions: Drs Dudzinski and Weiner had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Dudzinski, Bhatia, Isselbacher, Picard, Weiner.
Acquisition, analysis, or interpretation of data: Dudzinski, Bhatia, Mi, Picard, Weiner.
Drafting of the manuscript: Dudzinski, Weiner.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Dudzinski, Bhatia, Mi, Weiner.
Administrative, technical, or material support: Dudzinski, Bhatia, Isselbacher Picard, Weiner.
Study supervision: Dudzinski, Bhatia, Picard, Weiner.
Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported.