Prevalence of Financial Considerations Documented in Primary Care Encounters as Identified by Natural Language Processing Methods

Key Points Question How often are cost considerations documented in narrative clinical notes in a primary care setting? Findings In this cohort study of a data set including 222 457 outpatient primary care notes for 46 244 patients at a large academic medical center, 13.1% of patients had at least 1 note indicating a financial conversation with their physician. Specific socioeconomic features were associated with the presence of documented cost considerations. Meaning Although the literature suggests that patients desire to discuss health costs with their physicians, these conversations remain infrequent.


Introduction
Household out-of-pocket spending on health care has been increasing steadily over recent years in the United States, driven at least in part by the growing popularity of high-deductible health plans. 1 In response, physicians are becoming increasingly mindful of patient spending: some physician groups explicitly advocate for the consideration of costs in their clinical guideline documents, 2  recommendations. There is also a growing body of literature examining the prevalence and content of cost-related conversations between patients and physicians. Most US individuals, up to 70% according to a large national study from 2017, 4 are interested in discussing cost with their physician, although current estimates of the actual prevalence of cost conversations vary widely, from 4% to 65%. 5 These previous estimates largely relied on either survey data, which may be biased, or recorded clinic interactions, which may result in the disruption of care.
As an alternative to these existing methods, in our previous work, 6 we applied machine learning to electronic health records to develop a highly discriminative model identifying the presence and nature of financial considerations in intensive care unit (ICU) clinical notes. Here, we sought to understand the prevalence of such conversations in a cohort likely to be more representative of medicine as a whole, particularly outpatient medicine. We applied the model trained on the ICU data to outpatient primary care notes from a large academic medical center. As in the ICU setting, we also aimed to understand the extent to which these conversations might be associated with patient-level sociodemographic features.

Model Training
We have previously described the derivation of the natural language processing (NLP) model used to classify notes. 6 In brief, the notes labeled for the presence of financial conversations from the previous study were used to train a random forest classifier with Python's scikit-learn package (sklearn.ensemble.RandomForestClassifier, version 0.20.0). 9 These notes were randomly split into a training set of 5021 of 5579 notes (90.0%) for model development and tuning, and a testing set of 558 (10.0%) was used to confirm the model's ability to generalize. The number of notes used to train the model was slightly higher than that in the previous study because nonindex admission notes were also included to allow for more data to train on. As previously discussed, 6 standard NLP techniques, including removing punctuation and English-language stop words, stemming, generating unigrams and bigrams, creating a term frequency-inverse document frequency matrix from these unigrams and bigrams, and using principal components analysis to reduce the dimensionality of this matrix, were used to transform the text before model training.
Areas under the curve calculated by using the trapezoidal rule, precision, and recall were used as the primary evaluation metrics for discrimination on the training and test sets. On the independent testing set, the area under the curve exceeded 0.96. Precision and recall were calculated at different

Model Evaluation on Outpatient Data Set
The 222 457 clinical notes in the outpatient data set were preprocessed in exactly the same way as the training set. To validate the performance of the model on the outpatient data set, 1 author manually tagged 100 of these notes to determine a reference standard. On these 100 notes, the area under the curve was 0.83, and precision and recall were 0.83 and 0.93, respectively, at the predetermined 0.70 threshold. One author then manually tagged 300 patient notes identified by the model as having a documented financial conversation into further subcategories according to the nature of the conversation.

Statistical Analysis
Summary statistics, including proportion for categorical features and mean (SD) for continuous features, were computed for each group (ie, patients who had engaged in financial conversations and patients who had not) using Python's pandas (version 0.23.4) 10 and scipy (version 1.1.0) 11 packages.
No data were missing. Comparisons used χ 2 tests or single-sample, unpaired, 2-tailed t tests, as appropriate. After these univariate analyses, binomial logistic regression models using Python's statsmodel (version 0.9.0), 12 adjusted for the total number of notes associated with the patient, age, sex, race/ethnicity, insurance type, and zip code median income at earliest recorded visit, were fit with the presence or absence of financial discussion as the dependent variable to estimate the independent effect of each sociodemographic feature in terms of odds ratios (ORs) and 95% confidence intervals. All reported P values are 2-sided, with nominal significance considered to be uncorrected P < .05.

Results
The data set as a whole included 222 457 outpatient notes for 46 244 patients seen between January

Discussion
In application of a previously validated NLP model to a data set of 222 457 outpatient primary care notes, we found that 13.1% of 46 244 patients had at least 1 documented conversation with their physician that referenced financial considerations. The percentage of patients with at least 1 costrelated conversation in a given year was fairly constant at 6.7% to 8.2% per year. In adjusted analyses, these discussions were more likely for unmarried individuals, individuals residing in areas with lower median household incomes, individuals with government health insurance, and individuals of black and Hispanic race/ethnicity.
Estimates of the prevalence of financial conversations vary widely across the literature; for a summary, see the article by Hunter et al. 5 These estimates include 30% of dialogue transcripts from specialist outpatient visits with a mention of cost, 1 15% of primary care physicians in California reporting discussing costs with their patients most or all of the time, 13 and 40% of African American Of note, all prior work on this subject is based on either survey data or recorded dialogue from clinic visits. To our knowledge, this study is the first to use NLP to arrive at an estimate of the prevalence of financial conversations in the outpatient setting. That our estimate falls within the range of previously published work not only validates the accuracy of this method but also illustrates how machine learning can be applied to large data sets to answer questions more efficiently and cost-effectively than ever before. Furthermore, we demonstrate how an existing model, in this case an NLP model trained on the ICU notes from our previous work, 6 can be repurposed for a second task. This strategy, known as transfer learning, can overcome issues in which a clinical data set is too large to be relabeled entirely (or too small for a model to be trained on it effectively). 15 The strong performance of the ICU model on the outpatient visit notes suggests that this model can be successfully applied to other data sets in the future.
In a random sample of notes with documented financial conversations, we found that 48.7% of conversations resulted in a change to treatment and 28.8% resulted in a change of medication. The finding that patient-physician discussions of cost can result in a change of management is in line with other research suggesting that physicians are increasingly incorporating patient out-of-pocket spending into their decisions-making. 2,3,16 Whether these changes are associated with health outcomes remains to be determined.
Unsurprisingly, our results demonstrate that socioeconomic factors are associated with the presence of cost conversations. Single patients, individuals residing in zip codes with lower median incomes, and those with government insurance including Medicaid likely experience a greater economic burden associated with medical care compared with other groups and, thus, are more likely to discuss cost with their physician. These findings may inform physician behavior and training; for example, medical students and residents may be taught which medications are more or less likely to be approved by public insurance plans. However, the association between Medicare insurance and the presence of financial notes is more surprising, because Medicare is intended to protect senior citizens against medical expenditure risk. We postulate that this may be reflective of global survey findings 17 showing that US seniors face more financial barriers to care, despite nearly universal access via Medicare, than their peers in other high-income countries. It may also reflect a decline in financial literacy among seniors, which may increase economic anxiety in this group. 18 We also found that black and Hispanic individuals are more likely to discuss cost with their physicians, as evident in both crude and adjusted models.

Limitations
We note multiple limitations in this work. Our model was initially trained on inpatient ICU notes, which may involve language and themes that are different from those of outpatient settings. It is also likely that we underestimate the true prevalence of patient-physician financial conversations because some discussions may not be documented in clinical notes or may take place with front desk or billing staff instead of the physician. The present data also do not allow investigation of physicianlevel features that likely are associated with the probability of such discussions. In addition, other unmeasured variables (eg, medical comorbidities and the presence or absence of an acute problem at time of visit) are likely to be associated with the probability of financial discussion. Also, our analysis was limited to a single academic medical center; future applications of the model to data from other sites may reveal interesting regional and institutional differences in the presence of cost conversations.