Proportions are accompanied by Wilson score 95% confidence intervals (error bars). NE indicates no echocardiography; TEE, transesophageal echocardiography alone; TTE, transthoracic echocardiography alone; TTE+TEE, both TTE and TEE; TEE|TTE−, TTE first, followed by TEE only if the TTE result is negative for endocarditis; and TEE|TTE+, TTE first, followed by TEE only if the TTE result is positive for endocarditis.
eMethods. Supplemental Methods
eFigure 1. Distribution of S aureus Bacteremia Scenarios by Variables of Interest
eTable 1. Negative Likelihood Ratios (95% Confidence Intervals) of the Six Echocardiography Strategies by Degree of Error Covariance Between Transthoracic and Transesophageal Echocardiography
eTable 2. Positive Likelihood Ratios (95% Confidence Intervals) of the Six Echocardiography Strategies by Degree of Error Covariance Between Transthoracic and Transesophageal Echocardiography
eResults. Supplemental Results
eFigure 2. Preferred Echocardiography Strategy by Scenario VIRSTA Score
eTable 3. Results of Univariate Analyses for the Recommendation of a Highly Exclusionary Echocardiography Strategy
eTable 4. Multivariate Complete Case Analysis for the 4334 Recommendations From Respondents Providing an Estimate of Their Confidence in Their Estimated Risk of Relapse of S aureus Endocarditis After Short Course Therapy
Customize your JAMA Network experience by selecting one or more topics from the list below.
Identify all potential conflicts of interest that might be relevant to your comment.
Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.
Err on the side of full disclosure.
If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.
Not all submitted comments are published. Please see our commenting policy for details.
Heriot GS, Tong SYC, Cheng AC, Liew D. A Scenario-Based Survey of Expert Echocardiography Recommendations for Patients With Staphylococcus aureus Bacteremia at Varying Risk for Endocarditis. JAMA Netw Open. 2020;3(4):e202401. doi:10.1001/jamanetworkopen.2020.2401
Which echocardiography strategies do expert clinicians recommend for patients with Staphylococcus aureus bacteremia at varying risk for endocarditis?
In this survey study of 656 respondents from 24 countries, the echocardiography recommendations of expert clinicians for a variety of S aureus bacteremia scenarios were closely aligned to the burden of endocarditis risk factors. Substantial disagreement was seen between respondents as to the single best echocardiography strategy, particularly for scenarios with a low risk of infective endocarditis.
Experts demonstrate active and precise risk stratification when recommending echocardiography strategies for patients with S aureus bacteremia, but disagreement exists as to the appropriate investigation of lower-risk cases.
Echocardiography to detect infective endocarditis is regarded as a key quality indicator in the care of patients with Staphylococcus aureus bacteremia, but its application varies markedly between reported series. Understanding the reasons for this variation in practice is important to improve the use of this investigation.
To identify expert clinicians’ preferred echocardiography strategy for a variety of S aureus bacteremia scenarios in a hypothetical setting free from extrinsic constraints.
Design, Setting, and Participants
Anonymous web-based survey study comprising 50 text-based scenarios describing patients with S aureus bacteremia and various combinations of risk factors for endocarditis. Other variables included patient age and the presence of an extracardiac focus of infection warranting prolonged treatment. The survey was emailed to participants between September 2018 and March 2019. Each respondent was asked to recommend 1 of 6 echocardiography strategies for up to 8 randomly selected scenarios. Respondents were primarily infectious diseases physicians, and more than half reported an annual caseload of more than 20 cases of S aureus bacteremia.
Main Outcomes and Measures
The proportion of respondents selecting each of the 6 echocardiography strategies was calculated alongside Wilson score confidence intervals. Modified Fleiss κ statistics were used to described interrespondent variability. Generalized estimating equations were used to assess the associations between respondent- and scenario-level variables and the recommendation of an echocardiography strategy with a low negative likelihood ratio for infective endocarditis (ie, a highly exclusionary strategy).
A total of 656 respondents from 24 countries provided 4837 echocardiography recommendations across the 50 scenarios. Echocardiography recommendations were associated with scenarios’ burden of endocarditis risk (multivariate odds ratio per point of the VIRSTA score, 1.4; 95% CI, 1.4-1.5; P < .001). Poor interrespondent agreement was seen across all scenarios (modified Fleiss κ, 0.06; 95% CI, 0.05-0.07) but was most notable for scenarios with a lower risk of endocarditis (modified Fleiss κ, 0.04; 95% CI, 0.03-0.05). The presence of an extracardiac focus of infection was also associated with the choice of echocardiography strategy (odds ratio for highly exclusionary strategy, 0.51; 95% CI, 0.45-0.58). Respondent location in continental Europe was associated with recommendations in favor of a highly exclusionary strategy (odds ratio, 1.8; 95% CI, 1.3-2.5) compared with location in Australia or New Zealand.
Conclusions and Relevance
In this study, expert clinicians demonstrated active stratification by risk of endocarditis when making echocardiography recommendations for hypothetical patients with S aureus bacteremia. Substantial disagreement existed as to whether patients at lower risk of endocarditis should undergo transesophageal echocardiography–based echocardiography strategies.
Echocardiography is currently recommended for every patient with Staphylococcus aureus bacteremia (SAB) owing to the risk of clinically occult endocarditis and its associated complications.1,2 Within this broad recommendation, there appears to be significant variation in the application of echocardiography, even in series examining infectious diseases consultations3 and surveys of the recommendations of expert practitioners.4 In particular, the application of transesophageal echocardiography (TEE) appears highly variable, which may not be adequately explained by differences in patients’ risk factors for endocarditis.5 When asked, clinicians appear to identify endocarditis risk, patient frailty, and the presence of alternative foci requiring prolonged treatment as reasons to omit TEE for patients with SAB.6
In recent years, a number of publications have described tools designed to confer a numerical risk of endocarditis prior to the performance of echocardiography. Most of these have focused on patients at low risk of endocarditis,7 but some capture a range of risks, such as the VIRSTA score.8 This scoring system was derived from a multicenter French cohort study including more than 2000 episodes of SAB and used modified Duke criteria consensus adjudication at 12 weeks as the reference standard. Ten clinical and microbiological factors were independently associated with a diagnosis of endocarditis: C-reactive protein level greater than19 mg/dL (to convert to milligrams per liter, multiply by 10), severe sepsis, nonnosocomial acquisition, vertebral osteomyelitis, persistent bacteremia beyond 48 hours, intravenous drug use, preexisting native valve disease, permanent intracardiac device or previous endocarditis, meningitis, and cerebral or peripheral emboli. Although not externally validated for its predictive value, the VIRSTA score provides a comprehensive and substantiated basis on which to construct hypothetical scenarios focused on the diagnosis of endocarditis.
In this study, we aimed to explore the association of patient and clinician characteristics with the echocardiography recommendations of expert clinicians for a range of SAB scenarios in a hypothetical resource-unconstrained setting.
The study protocol and survey instrument were approved by the Monash University human research ethics committee. Respondents indicated their consent to participate by checking a mandatory radio button on the survey information page. This study followed the American Association for Public Opinion Research (AAPOR) disclosure checklist.
We developed an anonymous web-based multiple-choice survey (eResults in the Supplement) to assess the echocardiography recommendations of expert clinicians for a set of 50 text-based SAB scenarios based on the VIRSTA score.8 Scenarios were generated by varying patient age, the risk factors for endocarditis contained in the VIRSTA score, and the presence of an extracardiac focus of infection requiring prolonged treatment (either a prosthetic joint infection or vertebral osteomyelitis) to generate clinically plausible text-based descriptions of a SAB episode (eFigure 1 in the Supplement). Respondents were not informed that the scenarios were based on an endocarditis prediction score, nor that the scenarios were constructed specifically to vary their risk of endocarditis.
Although the VIRSTA score includes the presence or absence of intracardiac prosthetic devices (prosthetic valves or cardiac rhythm management devices), the presence of these devices has implications beyond an increased risk of endocarditis, affecting the diagnostic performance of echocardiography (particularly transthoracic echocardiography [TTE]), the chance of medical cure, and the development of other surgical indications. To aid interpretability, we chose to present only scenarios without these devices. For similar reasons related to excessive complexity of interpretation, we also chose not to include scenarios with symptomatic cerebral emboli or S aureus meningitis.
Each respondent was presented with 8 scenarios randomly selected without replacement from the overall pool of 50 scenarios: 2 of 15 very low-risk scenarios (VIRSTA score ≤2), 3 of 16 low-risk scenarios (VIRSTA score 3 or 4), and 3 of 19 moderate- to high-risk scenarios (VIRSTA score ≥5). This selection process ensured that each respondent addressed a range of endocarditis risk and emphasized the low- and very low-risk scenarios where we anticipated the greatest variability in responses.
Respondents were advised to consider the following conditions as applying to each scenario: TTE and TEE were equally accessible; no specific contraindication to TEE was present; there were no clinical features of an intracardiac complication of endocarditis (eg, heart failure or atrioventricular conduction disturbance); and no new complications of SAB arose unless stated.
For each scenario, respondents were asked to select their preferred echocardiography strategy from 6 options, as if they had been asked to provide a recommendation on day 3 to 5 after the onset of bacteremia. The 6 echocardiography strategies were TEE as the only study; both TTE and TEE; TTE first, followed by TEE only if the TTE result is negative for endocarditis; TTE first, followed by TEE only if the TTE result is positive for endocarditis; TTE as the only study; and no echocardiography. The first 3 strategies all have a negative likelihood ratio for native valve endocarditis of less than 0.10 (eTable 1, eTable 2, and eMethods in the Supplement) and are subsequently referred to as highly exclusionary echocardiography strategies. No information regarding the diagnostic performance of any echocardiography strategy was provided to respondents.
Respondents were also asked to provide nonidentifying demographic details including country of clinical practice, medical specialty, years of postfellowship experience, and annual SAB caseload. We also sought each respondent’s assessments of the risk and consequence of relapse after inadvertent treatment of clinically occult S aureus native valve infective endocarditis with short-course antibiotic therapy.
The survey was initially piloted among a small group of local and international contacts who provided feedback on the number and framing of survey questions and the appropriateness of the presented echocardiography strategy descriptions. Once finalized, the survey was distributed via the email lists of various national specialty societies and via direct email contacts between September 2018 and March 2019. The survey was made available to respondents in 3 languages (English, French, and Spanish). Overall response rates or predictors were not calculable, but list-specific response rates were estimated for lists with known distribution numbers.
Data were primarily analyzed using descriptive statistics. Wilson score 95% confidence intervals were calculated for all proportions. Interrater agreement across the 6 response categories was assessed using Fleiss κ modified for nominal data using the Monte Carlo simulation package raters in the R statistical environment (R Project for Statistical Computing). Associations between the recommendation of a highly exclusionary strategy and respondent demographic data, the number of VIRSTA points, patient age, and the presence of an extracardiac focus of infection were assessed using a scenario-level model (generalized estimating equations, binomial response, logit link, exchangeable correlation structure, and robust standard errors) clustered by respondent using STATA version 14.2 (StataCorp). Statistical significance was set at 2-tailed P < .05.
During the survey period, 656 respondents from 24 countries provided echocardiography recommendations for at least 1 survey scenario, giving a total of 4837 echocardiography recommendations (median [interquartile range], 96 [85-106] responses per scenario). Most respondents were infectious diseases physicians, and more than half reported an annual caseload of more than 20 cases of SAB (Table 1). Response rates were estimable for the mailing lists of the Australasian Society of Infectious Diseases (255 complete and partial responses from roughly 700 clinician members [approximately 35%]) and the Emerging Infections Network (204 complete and partial responses from roughly 1300 physicians with an adult infectious diseases practice [approximately 15%]).
There was little consensus among respondents as to the preferred echocardiography strategy in any of the 50 scenarios. Poor interrater agreement was evident across the 6 response categories, with a modified Fleiss κ statistic of only 0.06 (bootstrapped 95% CI, 0.05-0.07). In only 1 scenario was a single echocardiography strategy chosen by at least 50% of respondents: a patient aged 91 years with brief nosocomial SAB without sepsis or any clinical focus of infection, for whom 45 of 90 respondents (50%) preferred the strategy of TTE as the only study. Every 1 of the 6 strategies was selected by at least 1 respondent in 44 of the 50 scenarios (the strategy of no echocardiography was the only omission in 4 of the remaining scenarios).
Although disagreement persisted, trends in echocardiography recommendations were evident when scenarios were grouped by VIRSTA score. Highly exclusionary echocardiography strategies were increasingly favored as the number of risk factors for endocarditis increased (multivariate odds ratio per point of the VIRSTA score, 1.4; 95% CI, 1.4-1.5; P < .001). (Figure; eFigure 2 in the Supplement). Scenarios describing patients with a VIRSTA score of 5 or greater (roughly corresponding to a risk of endocarditis >10%) attracted a recommendation in favor of 1 of these highly exclusionary strategies in 1418 of 1736 responses (82%; 95% CI, 80%-83%). In contrast, recommendations were roughly evenly divided for lower-risk scenarios. For scenarios with a VIRSTA score of 5 or less, 1611 of 3101 recommendations (52%; 95% CI, 50%-54%) were in favor of a highly exclusionary strategy (P < .001 for the comparison). Interrater agreement at the level of the individual scenario was lower for these lower-risk scenarios (modified Fleiss κ, 0.04; 95% CI, 0.03-0.05) compared with the higher-risk scenarios (modified Fleiss κ, 0.08; 95% CI, 0.07-0.10).
The results of the multivariable analysis for the binary outcome of the recommendation of a highly exclusionary strategy are presented in Table 2. In addition to the association with VIRSTA score, this analysis also identified the presence of an extracardiac focus requiring prolonged treatment as having an independent association with recommendation against a highly exclusionary strategy (odds ratio, 0.51; 95% CI, 0.45-0.58). This negative association was stronger for prosthetic joint infection (odds ratio, 0.30; 95% CI, 0.26-0.35) than for vertebral osteomyelitis (odds ratio, 0.65; 95% CI, 0.56-0.76). Weaker associations were observed for respondent demographic characteristics. Respondent location in continental Europe was associated with recommendation of highly exclusionary strategy (odds ratio, 1.8; 95% CI, 1.3-2.5) compared with location in Australia or New Zealand. Annual caseload and years of postfellowship experience were not associated with choice of an exclusionary strategy on univariate analysis and were not included in the multivariable analysis (see eTable 1 in the Supplement). The associations between individual components of the VIRSTA score and recommendations in favor of a highly exclusionary strategy were closely aligned with the points value assigned to these components in the score’s original description (Table 3).
Significant missing data were observed for the questions dealing with the risk and consequence of relapsed endocarditis, with 15% and 18% of respondents not providing responses for these questions, respectively. Of these, the only variable noted to have an independent association with recommendations in favor of an exclusionary strategy was respondents’ reported confidence in their estimate of the risk of relapse after inadvertent short-course therapy, and then only for the 11 respondents who reported being “very confident” in their assessment (eTable 3 and eTable 4 in the Supplement).
Of note, respondents’ estimated risk of relapse after inadvertent short-course antibiotic therapy for S aureus endocarditis was not associated with recommendations in favor of highly exclusionary echocardiography strategies (odds ratio per 10% increase in estimated risk of relapse, 1.03 [95% CI, 0.99-1.07]; P = .12).
The 2 major findings of this survey are (1) that expert clinicians appear to engage in active and precise stratification by risk of endocarditis when making echocardiography recommendations for patients with SAB and (2) that there is considerable disagreement between clinicians as to the single preferred echocardiography strategy and the need for a highly exclusionary strategy among lower-risk patients.
A number of previous studies have presented mechanisms for the estimation of endocarditis risk in SAB,7 but there has been little prior research into whether clinicians can or do use this form of risk estimation to inform relevant clinical decisions such as the use of echocardiography. Although we did not point out to respondents that the clinical scenarios were based on a risk-stratification scoring system or provide any numerical score or estimate of endocarditis risk, respondents nevertheless appeared to alter their echocardiography recommendations in a very finely graded (and arguably appropriate) fashion. This result not only supports the relevance of endocarditis risk stratification to the management of SAB but also the ability of (expert) clinicians to implement this type of assessment in their clinical decisions.
While clinicians did appear finely attuned to endocarditis risk when analyzed in aggregate, there was sizeable disagreement as to the most appropriate choice of echocardiography strategy for any given scenario. The variation in recommendations was particularly marked for scenarios describing patients with a VIRSTA score of less than 5 points (corresponding to an endocarditis risk of <10%), for which recommendations were evenly divided between highly exclusionary strategies and those less able to exclude the diagnosis of endocarditis.
This variation in (hypothetical) practice mirrors previous survey and cohort data examining the use of echocardiography in SAB. Studies of routine infectious diseases consultation have reported that the application of TEE varies from 5% to 46% of patients with SAB treated by infectious diseases physicians,9 although these reports did not control for clinical differences between the included cohorts. Studies of endocarditis risk factors in SAB have suggested that clinicians do not pursue TEE nearly as often in patients with few risk factors for endocarditis as they do for patients at higher risk,7 but a previous multicenter cohort study5 found that the variation in the use of echocardiography between 3 geographically linked hospitals was inadequately explained by the prevalence of major risk factors for endocarditis. Such clinician-driven variation is also reflected in a recent survey of North American infectious diseases physicians,10 in which 20% of respondents indicated that they would routinely pursue TEE after a negative TTE result (regardless of the clinical circumstance), with the remainder selecting patients for follow-up TEE depending on unstated clinical factors.
Through the use of fixed hypothetical scenarios, our survey allows some separation of patient- and clinician-driven reasons for variation in echocardiography recommendations. Although we did not observe the association with years of postfellowship experience as echocardiography practices seen in a recent US survey,10 our results suggest that respondent characteristics may have some impact on these recommendations. Some practice locations and medical specialties were associated with the likelihood of a highly exclusionary strategy being recommended, although our ability to explore these associations was limited by the diversity of our respondents. Regional differences in recommendations also suggest that variation is not truly at the level of the individual clinician, but that local norms and health service resources are influential. Although we specifically asked respondents to imagine a setting where TEE and TTE were equally accessible, it is possible that habits shaped by local echocardiography availability contributed to this association.
Surprisingly, respondents’ estimates of the likelihood and consequence of relapse in the event of inadvertent treatment of S aureus native valve endocarditis with short-course therapy was not clearly associated with recommendations in favor of highly exclusionary echocardiography strategies. These variables have been shown previously to have an important role on the effect of the choice of echocardiography strategy on modeled 90-day mortality after SAB.11 The lack of association between echocardiography recommendations and the estimated consequence of relapse on overt questioning stands at odds with the strong association seen with the presence of an extracardiac focus requiring prolonged treatment in scenario descriptions. Prolonged treatment with regimens used for the specific extracardiac foci described in these scenarios would be expected to be generally effective for the treatment of uncomplicated native valve endocarditis12,13 and thereby largely obviate the relapse risk. The identification of these foci in patients with SAB has been previously identified as a reason clinicians choose not to pursue TEE.6 Of the 2 described extracardiac foci, prosthetic joint infection (a focus not associated with increased risk of endocarditis14) had a stronger negative association with the choice of a highly exclusionary strategy on univariate analysis than did the presence of vertebral osteomyelitis.
Our study has a number of limitations that arise from the design and distribution of the survey instrument. The scenarios presented to respondents are clearly simplifications of a complex disease process. Although we included endocarditis risk factors identified in the analysis of the largest relevant published data set,8 there may be other important pieces of clinical information respondents may use to guide their echocardiography decision-making. This may include whether a patient would be fit for cardiac surgery, a complex consideration that was not captured in our scenarios other than through the variables describing patient age and health care–associated acquisition of bacteremia. We also deliberately did not include scenarios describing patients with intracardiac prosthetic devices owing to the differences in the diagnostic performance of echocardiography, incidence of perivalvular complications, and relapse rate compared with patients without these prostheses. It is also worth reiterating that, although consistent with a large observational literature, the VIRSTA score has not been externally validated. In this study, we have used it primarily as a risk stratification tool and a source of relevant clinical parameters rather than a numerical risk calculator.
The echocardiography strategies from which respondents could choose also simplify the interpretation of multipart echocardiography results into a binary framework. Additional considerations, such as the image quality of TTE and its identification of nondiagnostic structural abnormalities, may be relevant for echocardiography decision-making.15,16
Furthermore, the distribution mechanism and estimated response rates to the survey make assessment of its representativeness uncertain. Our results are perhaps best understood as the opinions of an interested and motivated subgroup, which nevertheless contains substantial disagreement as to the most appropriate echocardiography strategy for various SAB scenarios. The opinions of cardiologists were clearly underrepresented in our results owing to the low number of responses. As with any survey posing hypothetical scenarios, it is also possible that the collected responses are not representative of the real-world practice even of those who provided responses.
This survey found that expert clinicians engage in active and precise risk stratification by risk of endocarditis when selecting an echocardiography strategy for patients with SAB free from availability constraints. Further studies are required to assess whether highly exclusionary echocardiography strategies are required for patients at lower risk of endocarditis, an area of substantial disagreement between respondents in this study.
Accepted for Publication: February 6, 2020.
Published: April 9, 2020. doi:10.1001/jamanetworkopen.2020.2401
Open Access: This is an open access article distributed under the terms of the CC-BY-NC-ND License. © 2020 Heriot GS et al. JAMA Network Open.
Corresponding Author: George S. Heriot, PhD, School of Public Health and Preventive Medicine, Monash University, Level 4, 553 St Kilda Rd, Melbourne, Victoria 3004, Australia (email@example.com).
Author Contributions: Dr Heriot had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: All authors.
Acquisition, analysis, or interpretation of data: Heriot, Tong.
Drafting of the manuscript: Heriot, Liew.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Heriot.
Obtained funding: Heriot.
Supervision: Tong, Cheng, Liew.
Conflict of Interest Disclosures: Dr Heriot reported receiving grants from Avant Mutual and grants from Australian Government Research Training Program during the conduct of the study. Dr Tong reported receiving grants from National Health and Medical Research Council during the conduct of the study. Dr Liew reported receiving grants from Abbvie, AstraZeneca, Bristol-Myers Squibb, Pfizer, Sanofi, Shire, and personal fees from Edwards Lifesciences outside the submitted work. No other disclosures were reported.
Funding/Support: This work was supported by an Avant Mutual Doctor in Training Research Scholarship (to Dr Heriot); by the Australian Government Research Training Program and the Monash University Faculty of Medicine, Nursing and Health Sciences (to Dr Heriot); and by an Australian National Health and Medical Research Council Career Development Fellowship (1065736 to Dr Tong).
Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Meeting Presentation: This work was originally presented at the International Society for Cardiovascular Infectious Diseases meeting; June 3, 2019; Lausanne, Switzerland.
Additional Contributions: We thank the survey respondents who participated and the various specialist societies and international contacts who assisted with the distribution of the survey.
Create a personal account or sign in to: