Physician Self-assessment of Shared Decision-making in Simulated Intensive Care Unit Family Meetings

Key Points Question Does intensivist self-assessment of communication skills endorsed by professional guidelines align with assessments of the same skills by blinded expert colleagues? Findings In this survey, 76 US intensivists read deidentified transcripts from their intensive care unit family meeting and rated themselves as conveying prognosis, highlighting choice, providing recommendations and offering the option of care focused on comfort. Sixty-one of 76 intensivists reported conveying prognosis, and blinded colleagues agreed that 42 of those 61 had conveyed the patient’s risk of death. Meaning Clinicians who lack communication skills endorsed in professional guidelines may also lack the metacognitive skills required to recognize their deficiencies, which makes routine feedback and continuing education on communication and shared decision-making essential at all levels of practice.

encourage ICU physicians (intensivists) to practice shared decision-making regarding preferencesensitive interventions. 3 Determining how doctors are interpreting and implementing recommendations about shared decision-making in the ICU presents a challenge. When family meetings are recorded, analyses are usually performed by clinical experts who review written transcripts, audio recordings, or video of family meetings for research purposes. [4][5][6][7][8][9][10][11][12] This method of studying communication is efficient and practical but may not reflect how intensivists experience or interpret a conversation. For a recommendation or guideline to change practice, clinicians must either be able to independently determine whether they are following it or receive feedback on adherence. 13 To understand how intensivists self-evaluate their shared decision-making skills, we asked 116 attending intensivists from US hospitals to review the transcript of a simulated ICU family meeting they participated in as part of a randomized clinical trial. 14 Intensivists who agreed identified both if and how they performed key elements of shared decision-making for an ICU patient at high risk of death. We then characterized the text that intensivists self-identified as examples of key communication skills recommended by their professional society's policy on shared decision-making. 3 We also compared intensivist self-evaluation with the evaluations of blinded colleagues reviewing the same transcripts for research purposes.

Study Design and Participants
We conducted a survey study that followed up intensivists who had participated in the Simulated Communication with ICU Proxies (SCIP) trial (NCT02721810). 14  with the daughter however they felt was appropriate given the clinical scenario. The actor was instructed to portray a daughter who was passive, had low health literacy, and would accept physician recommendations. Encounter recordings were transcribed verbatim and deidentified. This study was approved by the Johns Hopkins Medicine Institutional Review Board. Completion of the survey served as informed consent. This study followed the American Association for Public Opinion Research (AAPOR) reporting guideline for survey studies.

Survey Design and Distribution
In January 2019, at least 1 year after they had participated in the original trial, all SCIP trial participants were recruited via email to complete a follow-up survey. The survey asked the trial participants to identify key elements of shared decision-making in their own transcribed family meeting from the SCIP trial. The elements were drawn from a joint policy statement issued by The American College of Critical Care Medicine and American Thoracic Society in 2016 on shared decision-making in ICUs. 3 The policy statement endorsed key communication skills including disclosing prognosis, highlighting choice, and providing a recommendation. Participants' self-evaluated performance of these guidelines was assessed via the following 3 survey questions: "During the simulation, did you (1) convey prognosis for risk of death? (2) highlight that there is a choice? and (3) provide a recommendation?" Additionally, the Critical Care Choosing Wisely Task Force recommends offering the option of care focused entirely on comfort for patients at high risk for death or severely impaired functional recovery. 17 To assess how intensivists implemented this recommendation, study participants were JAMA Network Open | Critical Care Medicine Physician Self-assessment of Shared Decision-making in Simulated Intensive Care Unit Family Meetings asked a fourth question: During the simulation, did you offer the alternative of care focused entirely on comfort? All 4 survey questions had the following response options: done, not done, or not applicable. If an intensivist marked a communication skill as done, they were instructed to mark all text in their transcript corresponding to the performed skill. Intensivists received $50 for returning their questionnaire and annotated transcript.
We excerpted the text marked in simulation transcripts in response to 2 of the survey questions: (1) During the simulation did you highlight that there is a choice? and (2) During the simulation did As part of the original SCIP trial analysis the deidentified transcripts from each simulated encounter were reviewed by 2 blinded nonparticipant intensivist colleagues. These colleagues evaluated with a yes or no response whether intensivists conveyed prognosis for risk of death during the simulated family meeting. The colleagues also evaluated the clarity with which intensivists offered the option of care focused on comfort with responses of not offered, not understandable, vague, understandable, or clear. Interreviewer differences were reconciled via discussion. 14 We compared these evaluations with intensivists' self-reported responses to the same questions collected via survey more than a year later.

Statistical Analysis
We used descriptive statistics to characterize enrolled participants and summarized deductively coded response categories using counts and proportions. To test whether self-assessment differed from assessments by blinded colleagues when deciding whether prognosis had been conveyed, we estimated an odds ratio using an exact version of the McNemar test 18,19 for paired binary response data because of small count size in some cells. A 2-sided P<.05 was considered statistically significant. Data were analyzed using R, version 3.6.0 (R Project for Statistical Computing).

Results
The SCIP trial included 116 participants from 17 states. 14 Follow-up surveys were completed and returned by 76 (66%) intensivists. The mean (SD) respondent age was 43.1 (8.1), 72% were male, 76% identified as white (non-Hispanic), and first residency was completed a mean (SD) of 13.7 (8.5) years prior to the simulation. There were no statistically significant differences in age, sex, race/ ethnicity, or years since training when comparing intensivists who participated in vs those who declined to participate in the follow-up study ( Table 1).

Qualitative Analysis of Highlighting Choice
Thirty-two intensivists (42%) responded yes to the question, "During the simulation, did you highlight that there is a choice?" and no to the question, "During the simulation, did you offer care focused on comfort?" Representative text marked as highlighting choice by these 32 intensivists is presented in Table 2. The most common choice intensivists offered involved code status (34%).
Intensivists commonly presented this choice by contrasting cardiopulmonary resuscitation (CPR) and natural death: "There's um… [a] couple of things we could do if his heart would stop beating. One is we could do CPR. And try to bring him back. The other option is to allow him to… allow him to go naturally if his heart would stop on its own." Less common choices included continuing or adding forms of life support (16%) such as dialysis: "…would he want dialysis, do you think?" Three intensivists ( 9%) marked unclear or ambiguous language as highlighting choice. For example, this excerpt, which is a question to the surrogate, was marked as highlighting the existence of a choice: "Has he ever talked to you about what to do if say,

JAMA Network Open | Critical Care Medicine
Physician Self-assessment of Shared Decision-making in Simulated Intensive Care Unit Family Meetings his heart were to stop and he was to need somebody to push on his chest to bring his heart back, has he ever talked to you about what he would want in such a situation?" One intensivist (3%) offered the choice of a palliative care consultation. No discernable choice was offered in the text marked by 14 intensivists (44%). However, most of these statements alluded that the daughter might be offered choices in the future if her father's clinical condition deteriorated further.

Qualitative Analysis of Providing a Recommendation
The 33 intensivists (43%) who responded yes to the question, "During the simulation, did you provide a recommendation?" identified a wide range of recommendations in their transcripts ( Table 3). The most common recommendation was to continue the current treatment plan (33%).
Intensivists either stated they would continue the current plan or recommended a treatment plan to maximize the patient's chances of survival: "I'd like to try to lighten the sedation and other things  Choice is unclear or ambiguous 3 (9) "And I just want you to know that we're going to continue to treat these things aggressively, but there might be point where you and the rest of your family think that we're doing too much, and if you ever think we're doing too much to him that's not going to benefit him, please let us know." Multiple options rather than 1 choice 2 (6) "… if he wasn't able to get off the ventilator… would he find it acceptable to have tracheotomy tube… a tube here instead of attaching the machine to the tube in his mouth… and whenever we do that, we have to be able to feed you… we would have to put a feeding tube in through the stomach in order to feed you… You know, you can't manage the machine at home… So that does mean going to a facility…" Care focused on comfort in the future 2 (6) "…you and I will sit down and have a discussion as to whether our care goals change, okay? And whether instead of trying to prolong his life and survival, whether we shift to making sure that he is absolutely comfortable." Consultation to palliative care team 1 (3) "We have people in the hospital whose specialty is making people comfortable and that doesn't mean that they can't do that also while a patient is still being treated with aggressive medical care with a goal of survival, they can work with us. They're called a palliative care team." No choice is offered yet 14 (44) "If he's not getting better or certainly if he's getting worse over the next few days or a week or so, we're going to certainly have to sit down again and decide how to deal with things." Abbreviations: CPR, cardiopulmonary resuscitation; DNR, do not resuscitate.

JAMA Network Open | Critical Care Medicine
Physician Self-assessment of Shared  Continue current treatment plan 11 (33) "And so I would recommend that we continue the treatments that we're using right now…" Withhold CPR 9 (27) "I strongly recommend to families that we avoid CPR…" Meet in the future to make decisions 8 (24) "Well, what's say we also make a plan to meet tomorrow maybe? And that way, after you chat with some of his friends and then family… you know… try to get a better sense of what he would've wanted." A specific therapy or procedure 4 (12) "We may need to put a catheter into either his neck or his groin for dialysis if the nephrology consultants feel that dialysis is indicated at this point." Provide or consult with palliative care 2 (6) "I also want to add the different doctors to the team… a palliative care doctor, if that's okay with you. It's basically a doctor who specializes in palliation in case things don't go well, in case we have to talk about end of life issues." Self-care for the surrogate 1 (3) "So, one of the things I like to tell my patients is you can't help take care of him unless you're also taking care of yourself." Comfort-focused care 1 (3) "I think if one of those things happened that it would… be aligned with what you've told me about him, for us to focus on comfort and make sure that he's suffering as little as possible, rather than continue to do more things to keep him alive while we." Unclear 12 (36) "He may not be able to know that they're here because we're keeping him comfortable with the sedation medicine and that's, of course, a very important goal that we have, so we don't want him to experience any discomfort or pain while we're treating this severe illness, but I think it would be important for the family." Abbreviation: CPR, cardiopulmonary resuscitation. significantly less likely to code that prognosis was conveyed compared with intensivists reviewing their own transcript (odds ratio, 0.10; 95% CI,0.01-0.44; P < .001).

Blinded Colleague Ordinal Assessment of Offering Care Focused on Comfort
Twenty-five intensivists (33%) reported offering comfort care during the simulated family meeting after reviewing their own transcript ( Table 4) Intensivists were also significantly more likely to self-report they conveyed risk of death when compared with blinded colleagues reviewing the same transcripts. In addition, one-third of intensivists reviewing their own transcript reported that they offered care focused exclusively on comfort, but blinded colleagues reading these transcripts disagreed with this assessment more than half the time.
Measuring shared decision-making is difficult and evaluating decision-making at the end of life is particularly challenging. Landmark studies assessing shared decision-making in the ICU 10 as well as how ICU clinicians discuss patient preferences and values 6,7 have used trained study personnel to deductively code transcripts of recorded conversations for the presence of specific topics and skills.
Instruments and methods for evaluating clinician-patient interactions such as the Observer OPTION 5 scale 12 and the Roter Interaction Analysis System, 11 are similar in that trained experts with high interrater agreement review audio-or video-recorded encounters and decide whether predefined acts occurred. Qualitative analysis using inductive coding by experts has also been used to develop theories about how intensivists respond to questions about prognosis, 20,21 statements about spirituality, 22 and empathize 23 during family meetings. Our results suggest that methods that ask doctors to self-evaluate shared decision-making face the same potential compromises to validity as patient-reported measures of shared decisionmaking, namely that physicians may not recognize the encounter as a decision point or agree that a decision exists and that encounters between clinicians and families of ICU patients often include more than one complex decision. 24 Moreover, methods of evaluating physician communication skills that rely on self-report are vulnerable to the Dunning-Kruger effect, in which people who lack a skill also lack the ability to evaluate competence-including their own competence-in that skill. [25][26][27] This effect may also prevent a clinician with underdeveloped communication skills from recognizing when they are not adherent with shared decision-making recommendations.
Almost half of intensivists in this study reported that they highlighted a choice during the simulation but did not offer the option of care focused on comfort. In about one-third of these transcripts, the intensivists explained that the surrogate might have a choice in the future if her father's condition deteriorated further. For example: "If he's not getting better or certainly if he's getting worse over the next few days or a week or so, we're going to certainly have to sit down again and decide how to deal with things." Similarly, one-third of intensivists who reported that they made a recommendation marked text in which they recommended continuing with current treatment, with another 24% recommending that the conversation be reinitiated in the future. This finding suggests that while critical care guidelines endorse providing the option of care focused on comfort, 17 intensivists may view these guidelines as applying only after patients have failed lengthy trials of intensive care. 28

Limitations
This study has limitations. First, 34% of invited intensivists chose not to participate in this follow-up survey. Second, disagreements between self-assessment and blinded colleague assessment could be attributable to the colleagues harboring unusual views and that interrater reliability was not assessed. Third, nonverbal communication is not reflected in a transcript. Fourth, intensivists may behave differently in simulations than in clinical settings. In addition, we asked intensivists rather than patient surrogates to assess what was communicated to a simulated surrogate with low health literacy. Patient surrogates are better qualified to report this outcome and would ideally be used as a reference standard in this scenario. However, using surrogates with varying degrees of health literacy to interpret lengthy transcripts of simulated encounters is expensive and logistically complicated.