Example from a run chart tracking contextual probing (Probe) and contextualization of the plan of care (POC) at site E. Probe is the percentage of red flags heard by coders that were probed by the physician; POC is the percentage of contextual factors heard by coders that were incorporated into the plan of care by the physician.
The term audit refers to the period during which baseline data was collected on contextualization of care rates (at sites C-F) with no feedback to clinicians.
Customize your JAMA Network experience by selecting one or more topics from the list below.
Identify all potential conflicts of interest that might be relevant to your comment.
Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.
Err on the side of full disclosure.
If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.
Not all submitted comments are published. Please see our commenting policy for details.
Weiner S, Schwartz A, Altman L, et al. Evaluation of a Patient-Collected Audio Audit and Feedback Quality Improvement Program on Clinician Attention to Patient Life Context and Health Care Costs in the Veterans Affairs Health Care System. JAMA Netw Open. 2020;3(7):e209644. doi:10.1001/jamanetworkopen.2020.9644
Can data from patient-collected audio recordings of their visits help clinicians improve their attention to patient life context when planning care, improve patient outcomes, and reduce health care costs?
In this quality improvement study, feedback was delivered to 666 clinicians based on analysis of 4496 audio-recorded visits. Attention to patient contextual factors increased from 67% to 72%, and contextualized care planning was associated with a greater likelihood of improved outcomes, resulting in an estimated cost savings of $25.2 million from avoided hospitalizations.
These findings suggest that continuous feedback to clinicians about their attention to patient life context, based on audio recordings of their care, may substantially improve their performance, with measurable benefits for their patients and substantial cost savings.
Evidence-based care plans can fail when they do not consider relevant patient life circumstances, termed contextual factors, such as a loss of social support or financial hardship. Preventing these contextual errors can reduce obstacles to effective care.
To evaluate the effectiveness of a quality improvement program in which clinicians receive ongoing feedback on their attention to patient contextual factors.
Design, Setting, and Participants
In this quality improvement study, patients at 6 Department of Veterans Affairs outpatient facilities audio recorded their primary care visits from May 2017 to May 2019. Encounters were analyzed using the Content Coding for Contextualization of Care (4C) method. A feedback intervention based on the 4C coded analysis was introduced using a stepped wedge design. In the 4C coding schema, clues that patients are struggling with contextual factors are termed contextual red flags (eg, sudden loss of control of a chronic condition), and a positive outcome is prospectively defined for each encounter as a quantifiable improvement of the contextual red flag. Data analysis was performed from May to October 2019.
Clinicians received feedback at 2 intensity levels on their attention to patient contextual factors and on predefined patient outcomes at 4 to 6 months.
Main Outcomes and Measures
Contextual error rates, patient outcomes, and hospitalization rates and costs were measured.
The patients (mean age, 62.0 years; 92% male) recorded 4496 encounters with 666 clinicians. At baseline, clinicians addressed 413 of 618 contextual factors in their care plans (67%). After either standard or enhanced feedback, they addressed 1707 of 2367 contextual factors (72%), a significant difference (odds ratio, 1.3; 95% CI, 1.1-1.6; P = .01). In a mixed-effects logistic regression model, contextualized care planning was associated with a greater likelihood of improved outcomes (adjusted odds ratio, 2.5; 95% CI, 1.5-4.1; P < .001). In a budget analysis, estimated savings from avoided hospitalizations were $25.2 million (95% CI, $23.9-$26.6 million), at a cost of $337 242 for the intervention.
Conclusions and Relevance
These findings suggest that patient-collected audio recordings of the medical encounter with feedback may enhance clinician attention to contextual factors, improve outcomes, and reduce hospitalizations. In addition, the intervention is associated with substantial cost savings.
Contextualizing care is the process of adapting research evidence to patient context.1,2 For example, recognizing that a patient has lost control of their diabetes because they cannot afford insulin glargine and switching them to a less costly alternative is a contextualized care plan. Conversely, the failure to adapt research evidence to patient context, when it results in an inappropriate plan of care, is termed a contextual error.3,4 An attempt to improve diabetes control by increasing the dosage of medication that a patient cannot afford, and therefore has not taken as directed, would represent a contextual error.
Contextual errors occur frequently,5,6 adversely impact health care outcomes,6 and can increase health care costs.7 Nevertheless, they are difficult to detect because they are rarely discoverable from reviewing the medical record. In the aforementioned example, the physician is likely to document in their note that they responded to the patient’s poor diabetes control by increasing their insulin glargine dosage, which would seem reasonable. They are not likely to document that the patient cannot afford the medication, because they never elicited that information.
A more reliable way to ascertain whether a care plan was contextualized is to audio record the encounter, in addition to reviewing the medical record of that encounter. An efficient method for collecting audio recordings of encounters is to invite patients to carry audio recorders into their visits.8 Encounters are then analyzed using a coding method called Content Coding for Contextualization of Care (4C),9 which has demonstrated 85% interrater agreement for differentiating a contextualized care plan from a contextual error across trained coders. Coders follow a 4-step process of looking for clues that a patient is struggling (ie, contextual red flags) and then listening for whether the physician asked about them (ie, contextual probes), whether relevant life circumstances were identified (ie, contextual factors), and whether there was an attempt to address them in the care plan.
In a study6 of approximately 600 encounters using 4C analysis, contextual red flags were more likely to resolve or improve at 4 to 6 months after an audio-recorded visit if the care plan had been contextualized. Examples of such context-specific good outcomes, taken from the 4C coding manual,10 are shown in Table 1. The study provided observational data on the benefits of contextualizing care.
On the basis of these data, in 2013 the Department of Veterans Affairs (VA) introduced a quality improvement (QI) program at 2 of the sites in this study in which physicians receive ongoing feedback on their performance at contextualizing care, tracking 2 metrics: the percentage of audio-recorded encounters in which contextual red flags were probed when present and the percentage of recorded encounters in which contextual factors were addressed in the care plan when present (Figure 1). Feedback to clinicians consisted of a 4-line summary of each recorded visit indicating the contextual red flag, the contextual probe (if any), the contextual factor (if any), and the contextualized or noncontextualized care plan (see examples in Table 2), and a run chart (Figure 1) showing the 2 tracked metrics trended over time. Feedback was offered at 2 levels of intensity, starting with standard feedback, consisting of monthly reports to physicians with anonymized case examples discussed at monthly meetings and a case-of-the-week emailed to all clinical staff, followed by the expansion of the program (enhanced feedback) to include nurses and clinical pharmacists in the same practice groups, residents when present, online reflective exercises for continuing medical education (CME) and board recertification credit based on the recorded cases, optional individualized reports, and inclusion of data on the outcomes of contextual red flags (Table 2).
We report here a quality improvement study designed to address the following questions11: Does a patient-collected audio program with audit and feedback of data on contextualization of care prevent contextual errors? If so, is the reduced error rate associated with improved health care outcomes 4 to 6 months later as defined by resolution or partial resolution of contextual red flags identified at the audio-recorded visit (Table 1)? Finally, how do the costs of the program compare with cost savings that can be attributed to the program? For the latter, we focused on emergency department visits and hospitalization rates of patients whose physicians participated in the program. We selected these because they are dichotomous, unambiguous outcome measures that reflect the effectiveness of outpatient care for many conditions and substantially drive costs.
This study was approved by the VA Central Institutional Review Board. The program was approved by the Quality Improvement Committee or comparable authority at each site. Patients who participated in the audio recordings provided written, informed consent. This study follows the Standards for Quality Improvement Reporting Excellence (SQUIRE) reporting guideline.
Ambulatory care clinics based at 6 VA medical centers participated in this QI study (Figure 2). All were primary care clinics, except at site E, where a gastroenterology clinic was included. The program had been intermittently operational at 2 of the sites (sites A and B) since 2013, depending on leadership support and funding. At site A, the program was active at the start of the study, with clinicians receiving enhanced feedback. At site B, clinicians had received standard feedback before the start date, but the program had been inactive for the prior 26 months. Sites C, D, E, and F had no prior exposure to the program and were added sequentially, as illustrated in Figure 2. After a 7-month audit-only period for the 4 new sites, sites C and D began standard feedback, whereas sites E and F remained audit only. At 6-month intervals, paired sites advanced until all were receiving enhanced feedback (at site B, pharmacists and nurses did not participate). Assignment of sites C through F, each a cluster, to the steps was randomized.
In each participating clinic, a project assistant and trained volunteers staffed a table in the waiting room with flyers and posters advertising the program. Patients who expressed interest were informed that they could audio record their visit using an encrypted audio recorder for a program designed to help clinicians improve the quality of patient care. They could either carry it out in the open, or conceal it if they preferred (at site B, patients were instructed to reveal the recorder per site-specific protocol), and turn it off at any time. Participants in the QI program signed a consent form and returned the audio recorder after their visit as they exited the clinic. The recordings were uploaded to a secure server, and a team of 4C-trained coders with access to the patient medical record coded the encounter, identifying contextual red flags, contextual probes, contextual factors, and contextualized care plans, where present. To determine the outcome, 4 to 6 months after each encounter, a member of the coding team looked in the patient’s medical record for the status of any identified contextual red flags and noted whether they had improved, using predetermined criteria in the 4C coding manual10 (Table 1). To avoid bias, the coder was blinded as to whether the care plan for the encounter had been coded as contextualized.
Before the initiation of standard feedback, at least 1 clinical champion was designated at each site. The clinical champion was a peer—typically a primary care physician—who facilitated the discussion of data generated from the 4C coding. Clinical champions received training on giving feedback either during a 1-day workshop at the start of the study, or through coaching sessions over the telephone. Approximately once per month, each clinical champion received an electronic slide presentation emailed by the coding team, with each slide summarizing the 4 essential elements of an audio recording (Table 2). Slides were organized in a consistent format starting with examples in which the physician failed even to probe a contextual red flag to those in which they successfully performed all needed steps to arrive at a contextualized care plan. Champions presented the slides and facilitated discussion at standing clinical meetings at their facility. In addition, the coding team posted a weekly case on the clinician’s listserve, alternating between examples of contextual errors and effectively contextualized care plans.
When it was time to introduce enhanced feedback, audio coders would begin listening to and coding sections of audio recordings that included interactions with nurses and pharmacists and adding slides of those interactions to the slide presentations. They also included data on the outcome for each example at the bottom of each slide (Table 2). After the feedback sessions, attending physicians were emailed an optional worksheet with questions to address related to 1 or more cases discussed, indicating changes they would make to their practice. Participation was awarded with both CME credit and maintenance of certification credit from the American Board of Internal Medicine. Physicians who requested it were sent examples from their own clinics. Feedback to residents was provided either by the clinical champion or someone else designated by the residency program director or chief resident.
The program implementation process was guided by 3 principles: that it not be burdensome to clinicians or patients, that it feel safe to all participants, and that they see value in participating. Hence, feedback occurred only during pre-existing standing meetings, data were deidentified by the coding team before sharing, and physicians received CME and board recertification credits where applicable. As with most operationally approved QI projects, nearly all clinicians were enrolled, with the exception of site B, where the QI committee authorized an opt-in program only.
Research tasks conducted for the purpose of assessing the effectiveness of the QI program consisted of analysis of data generated by the 4C coding and a budget analysis. The latter was conducted by maintaining a log of all staff time and other resources for all activities related to the program at all participating sites and centrally (eg, the 4C coding) for the purpose of tabulating expenses. Prorated salaries of all staff, including project assistants who handed out and collected audio recorders and the coders, were calculated and tabulated. Data on emergency department and inpatient rates for patients of physicians participating in the QI program were extracted from the VA Corporate Data Warehouse, using codes linking patients to their clinician.
We fitted mixed-effects logistic regression models to estimate the likelihood of (1) probing each red flag, (2) addressing each revealed contextual factor, and (3) improvement in presenting red flags after 4 to 6 months. Models included the step (baseline vs any feedback, standard vs enhanced feedback) as the primary factor and controlled for time in the study, study site, and hierarchal clustering of red flags and factors within patients and patients within clinicians. Sample size calculations based on prior findings suggested that a total of 408 recordings with contextual factors (and additional recordings with red flags but not factors) would provide 80% power to detect past effect sizes (eg, a change from 45% to 65% likelihood); hence, we aimed to collect 2040 recordings (408 per site for the 5 sites that were in each group) to be able to examine hypotheses within each site.
We estimated cost savings by fitting Cox proportional hazard models to the time from outpatient visit to first emergency visit or hospitalization, with step as the key factor and covariates for site and patient sex, race, age group, and risk (via the Nosos Score) and multiplying the estimated annualized difference in hospitalizations among the clinicians’ panels by the mean site-specific cost of hospitalization in 2018 dollars.12 Adjusted hazard ratios in the Cox models were tested using 2-sided z tests based on robust SEs, with P < .05 as the significance threshold. Analyses were conducted in R statistical software version 3.6 (R Project for Statistical Computing) and Stata statistical software version 15.1 (StataCorp). Data analysis was performed from May to October 2019.
From May 1, 2017, to May 22, 2019, 4496 recordings were made by patients across the 6 participating sites during visits with 666 clinicians (Table 3). Patients seen in the participating clinics had a mean age of 62.0 years of age, 92% were male, 68% were White, 21% were non-Hispanic African-American, and 6% were Hispanic/Latino. A total of 2994 audio recordings (67%) contained at least 1 contextual red flag. Among these, 6860 red flags were identified, of which 3768 were probed (55%), identifying 2164 contextual factors (57%). An additional 821 contextual factors were revealed by patients without probing.
Before analyzing the step effects, we determined whether the patient revealing the recorder during the visit was associated with clinician probing or planning using mixed-effects logistic regression models; there was no association with either probing (odds ratio [OR], 0.98; 95% CI, 0.87-1.11; P = .79) or planning (OR, 0.97; 95% CI, 0.80-1.19; P = .81). The likelihood of a clinician probing red flags increased with standard or enhanced feedback (adjusted OR, 1.5; 95% CI, 1.3-1.8; P < .001).
Before receiving feedback, clinicians addressed 413 of 618 contextual factors in their care plans (67%). After receiving either standard or enhanced feedback, they addressed 1707 of 2367 contextual factors (72%; OR, 1.3; 95% CI, 1.1-1.6; P = .01). When we controlled for probing, clinicians were more likely to contextualize the care plan in the enhanced than in the standard feedback condition (adjusted OR, 1.9; 95% CI, 1.4-2.7; P < .001).
At the medical record review 4 to 6 months after the visit, measured outcomes for contextual red flags were available for 1227 of the contextualized care plans, of which 901 (73%) were good, as defined by partial or full resolution of the contextual red flag (Table 1). Measured outcomes were available for 474 of the noncontextualized care plans, of which 218 (46%) were good. When we controlled for step and clustering in patients and clinicians, contextualized care planning was associated with improved outcomes (adjusted OR, 2.5; 95% CI, 1.5-4.1; P < .001).
In the budget analysis, the Cox regression models found no difference in emergency department visit rate among patients according to their clinicians’ exposure to feedback interventions, but found a significant decrease in the rate of hospitalization from 19.0% at baseline to 16.5% for patients seeing clinicians in the enhanced feedback step (adjusted hazard ratio, 0.93; 95% CI, 0.88-0.98; P = .03). As a result, we estimate that 987 hospitalizations were avoided across the 6 sites (approximately 2.5% of annual hospitalizations of patients of the clinicians), at a savings of approximately $25.2 million (95% CI, $23.9-$26.6 million) based on per-site hospitalizations avoided and per-site 2018 costs of an average hospitalization. The total cost of the intervention was $337 242.
Providing ongoing feedback to clinicians on their attention to patient life context based on data collected from audio recordings and the medical record may improve their performance at contextualizing care. Patient face challenges with refilling medications, making it to appointments, and understanding their treatment plans, among other things. When physicians see that their efforts to address these challenges are effective at resolving them, they may change how they practice. This change benefits patients by increasing the chances that contextual red flags (eg, not refilling medication or missing appointments) improve or resolve. Sites that participated in the enhanced feedback step had reduced hospitalization rates, with cost savings greater than the cost of the intervention.
It may not be surprising that when physicians take steps to address patient life challenges complicating care, those patients often benefit from the intervention. Contextualizing care may be regarded as the pragmatic effort of the clinician to address social determinants of health during the medical encounter.13 In their recent report13 on integrating social care on the delivery of health care, the National Academy of Medicine identified 5 complementary facilitating activities: awareness, adjustment, assistance, alignment, and advocacy. Two of them, awareness and assistance, correspond to identifying contextual red flags (eg, determining that a patient is missing appointments because of a transportation barrier, which is a contextual factor) and assisting in addressing them (eg, arranging for transportation vouchers, which is a contextualized care plan). The magnitude of the effect is substantial. Converted to a number needed to treat to improve 1 red flag, contextualizing care has a number needed to treat of approximately 6 (adjusting for site).14 Although contextualizing care is less efficacious than, for instance, adherence therapy, which consists of 7 home visits to increase patient medication adherence (number needed to treat, 2.2), it is also much less intrusive and costly.14
Giving clinicians ongoing feedback on their attention to the life challenges that their patients face may be an effective strategy for heightening their awareness of and attention to social determinants of health, which may greatly improve health care outcomes and reduce health care costs. The intervention’s success in changing clinician behavior is consistent with other studies indicating that audit and feedback is an effective strategy for influencing professional practice.15 From the clinician’s perspective, it is an opportunity to see one’s blind spots and adjust accordingly.
A limitation of this study was our inability to randomize 2 of the 6 sites into the stepped wedge because of their previous exposure to the QI program. As a result, 1 site only contributed to the enhanced feedback group, and the other, though contributing to all groups, was fixed in its position in the wedge. We controlled for site in all analyses to attempt to mitigate this limitation. A limitation of the budget analysis is that estimated cost savings are based on the panel population and phase of participation in the program, rather than on individual patient contextualized care results. Our data suggest that clinicians reduce hospitalization rates by improving care to their panel, but it is possible that other confounding influences were present.
Engaging patients to record their visits, when they are comfortable doing so, enables the collection of otherwise inaccessible data about attention to life context that may improve health care at a low cost. Our findings suggest that QI programs could be well advised to consider routine incorporation of training in contextualizing care through audit and feedback.
Accepted for Publication: April 25, 2020.
Published: July 31, 2020. doi:10.1001/jamanetworkopen.2020.9644
Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2020 Weiner S et al. JAMA Network Open.
Corresponding Author: Saul Weiner, MD, Department of Medicine, University of Illinois at Chicago, 820 S Damen Ave, M/C 103, Chicago, IL 60612 (email@example.com).
Author Contributions: Drs Weiner and Schwartz had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Weiner, Schwartz, Altman, Bartle, Weaver, Wopat.
Acquisition, analysis, or interpretation of data: Weiner, Schwartz, Altman, Ball, Bartle, Binns-Calvey, Chan, Falck-Ytter, Frenchman, Gee, Jackson, Jordan, Kass, Kelly, Safdar, Scholcoff, Sharma.
Drafting of the manuscript: Weiner, Bartle.
Critical revision of the manuscript for important intellectual content: Schwartz, Altman, Ball, Bartle, Binns-Calvey, Chan, Falck-Ytter, Frenchman, Gee, Jackson, Jordan, Kass, Kelly, Safdar, Scholcoff, Sharma, Weaver, Wopat.
Statistical analysis: Schwartz, Bartle.
Obtained funding: Weiner.
Administrative, technical, or material support: Weiner, Altman, Ball, Binns-Calvey, Chan, Frenchman, Jordan, Kass, Kelly, Safdar, Scholcoff, Sharma, Weaver, Wopat.
Supervision: Weiner, Altman, Gee, Weaver.
Conflict of Interest Disclosures: None reported.
Funding/Support: This study was funded by grant IIR 15-452 from the Department of Veterans Affairs, Office of Research and Development, Health Services Research and Development, and the quality improvement program received funds from Veterans Integrated Services Network 12.
Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Disclaimer: The views expressed in this manuscript are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs.
Additional Contributions: Beverly Gonzalez, PhD (Center of Innovation for Complex Chronic Healthcare, Edward Hines Jr VA Hospital, and Northwestern University Feinberg School of Medicine) provided assistance as a collaborator with the VA Contextual Audit and Feedback Study Group.
Create a personal account or sign in to: