Association of Low-Value Care Exposure With Health Care Experience Ratings Among Patient Panels

Key Points

Question  Do primary care professional (PCP) patient panels who receive more low-value care rate their health care experiences more favorably?

Findings  This quality improvement study of 100 743 PCPs with a mean of 258 patients each constructed a composite score of low-value care exposure for each PCP patient panel and estimated its association with patient ratings of health care. With 1 exception (waiting room time), all observed associations between low-value care exposure and health care experience ratings (overall health care, timely access to nonurgent care, timely access to urgent care, personal physician, and interactions with personal physician) were small and/or lacked statistical significance.

Meaning  This study did not find evidence that more low-value care for a PCP patient panel was associated with more favorable patient ratings of their health care experiences.

Importance  Patient reviews of health care experiences are increasingly used for public reporting and alternative payment models. Critics have argued that this incentivizes physicians to provide more care, including low-value care, undermining efforts to reduce wasteful practices.

Objective  To assess associations between rates of low-value service provision to a primary care professional (PCP) patient panel and patients’ ratings of their health care experiences.

Design, Setting, and Participants  This quality improvement study used Medicare fee-for-service claims from January 1, 2007, to December 31, 2014, for a random 20% sample of beneficiaries to identify beneficiaries for whom each of 8 low-value services could be ordered but would be considered unnecessary. The study also used health care experience reports from independently sampled beneficiaries who responded to the 2010-2015 Consumer Assessment of Healthcare Providers and Systems (CAHPS) Medicare fee-for-service survey. Statistical analysis was performed from January 1, 2019, to December 9, 2020.

Main Outcomes and Measures  The main outcomes were health care experience ratings from Medicare beneficiaries who responded to the CAHPS survey from 2 domains, namely “Your Health Care in the Last 6 Months” (overall health care, office wait time, timely access to nonurgent care, and timely access to urgent care) and “Your Personal Doctor” (overall personal physician and a composite score for interactions with personal physician). Beneficiaries in both samples were attributed to the PCP with whom they had the most spending. For each PCP, a composite score of low-value service exposure was constructed using the 20% sample; this score represented the adjusted relative propensity of the PCP patient panel to receive low-value care. The association between low-value service exposure and health care experience ratings reported by the CAHPS respondents in the PCP patient panel was estimated using regression analysis.

Results  The final sample had 100 743 PCPs, with a mean of approximately 258 patients per PCP. Only 1 notable association was found; more low-value care exposure was associated with more frequent reports of having to wait more than 15 minutes after the scheduled time of an appointment (a mean of 0.448 points lower CAHPS score on a 10-point scale for PCP patient panels who received the most low-value care vs the least low-value care). Although some other associations were statistically significant, their magnitudes were substantially smaller than those typically considered meaningful in other CAHPS literature and were inconsistent in direction across levels of low-value service exposure.

Conclusions and Relevance  This quality improvement study found that more low-value care exposure for a PCP patient panel was not associated with more favorable patient ratings of their health care experiences.

Patient-reported health care experiences are widely used to incentivize quality improvement through public reporting and performance-based payments. For example, the Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey measures of patient experiences are included in Medicare Advantage plan star ratings¹ and the Medicare Shared Savings Program accountable care organization contracts.² These ratings measure dimensions of care that are best reported by patients, such as physician communication and timeliness. They are empirically associated with other measures of clinical process, outcomes, efficiency, and safety.^3-5 A concern about this practice is that it may encourage physicians to provide more low-value services (care that is not associated with a clinical benefit), out of a belief that responding to patient demand or the perception that more care is better will improve their ratings.^6-9 This may lead to the wasteful use of health care resources and spending, possible iatrogenic injury, and limited success of alternate payment models such as accountable care organizations.¹⁰

In this study, we directly address the question: Do patients of physicians who provide more low-value care rate their health care experiences more favorably? (We use the terms receipt of, exposure to, or provision of low-value care interchangeably, regardless of how or by whom it was initiated.) More low-value care may be interpreted by patients as a signal of better or worse care depending on patient trust and preferences, as well as physician communication of rationales.^3,11-13 It remains unclear whether patients are advocating for low-value service provision.^10,11,14-17 However, as long as physicians may be acting out of a concern about patient dissatisfaction, it is imperative to address this question to gain physician buy-in to policies for curbing waste.

One study that garnered national attention found that patients who reported most favorably on their care and physician communication had a higher mortality risk, implying that catering to patient satisfaction may lead to worse outcomes.¹⁸ However, the additional attention that patients with severe illness receive may lead them to rate their access and communication with physicians more favorably than would healthier patients with fewer care needs. This scenario may induce a positive correlation between patient experiences and mortality that is actually a reflection of patient factors, not the associations of physician practices with patient experiences. The limitation of the study design that made it vulnerable to this potentially flawed inference is its reliance on data from the same unique patient for both the patient report and the outcome.^3,4,6,19

In the context of our research question, similar confounding due to patient characteristics can arise if such factors are associated with both receipt of low-value care and health care experiences. For example, some patients may have more opportunities to receive low-value care than other patients in similar states of health because they value health care more and thus seek care more frequently. Likewise, patients who comply with physician orders more often are likely to receive more services. Patients with such stronger preferences or adherence may also appreciate their care and physicians more. Consequently, receipt of low-value care may be associated with more favorable reports on care experiences from the same patients even when physicians’ low-value practice patterns are not associated with their patients’ care experiences, and even when patients do not differ between physicians.

We designed a study that eliminates this source of patient-level confounding. We used a 20% random sample of the full fee-for-service Medicare population to assess low-value care exposure for a primary care professional (PCP) patient panel and a much smaller, independent sample from the CAHPS survey to measure patient experiences. Because the 2 samples were independent and overlapped minimally, our analysis did not rely on observations about the same patients to assess associations. Although our study remains subject to unmeasured confounding arising from unobserved systematic sorting of patients to different PCPs, our methods permit stronger conclusions about the associations between physician provision of wasteful care and care experiences than approaches taken in prior studies.

To assess provision of low-value services, we analyzed claims and enrollment data from January 1, 2007, to December 31, 2014, for a 20% simple random sample of fee-for-service Medicare beneficiaries, with the first year serving as a look-back period for 2008 claims. We used hospital inpatient data, outpatient claims, and noninstitutional claims for services such as physician visits. For a given year, we required beneficiaries to be continuously enrolled in Medicare Parts A and B in that year (while alive) and the previous year. Eligible beneficiaries also had at least 1 claim for primary care services (Healthcare Common Procedure Coding System codes 99201-99215, G0402, G0438, and G0439) provided by a PCP, defined by specialty codes for general practice, family practice, internal medicine, or geriatric medicine (eAppendix 1 in the Supplement); we attributed beneficiaries to the PCP with whom they had the most spending during the year.²⁰ The research protocol was approved by institutional review boards at Harvard Medical School, the University of Chicago, and the National Bureau of Economic Research with a waiver of informed consent according to CFR §46.116 (e)(3)(ii) because the research could not practicably be carried out without the waiver.

We assessed patient experiences with data from the 2010-2015 fee-for-service Medicare CAHPS surveys. We assigned each CAHPS respondent to a PCP using the approach described but linked to a PCP in the prior year because the CAHPS survey is administered near the beginning of the year and asks about the prior 6 months. The overall CAHPS survey response rate was 41.9% during these years. The nonresponse rates to items in the CAHPS ranged from 3% to 67%, almost entirely owing to inapplicability of the item to a respondent who did not have relevant experience (eAppendix 2 and eTable 1 in the Supplement). Because the CAHPS sample was only 1.5% the size of the claims sample used to assess low-value service provision, the overlap between the 2 samples was minimal.

We excluded PCPs with fewer than 11 patients in the 20% sample (in compliance with our data use agreement with the Centers for Medicare & Medicaid Services) or without at least 1 attributed CAHPS respondent (eAppendix 3 and eFigure in the Supplement). Clinical covariates were extracted from the Chronic Conditions Data Warehouse,²¹ which tracks diagnoses of 27 conditions from 1999 or a beneficiary’s first year of Medicare enrollment onward, zip code–level sociodemographic data from the American Community Survey in the same years as our claims data, and data on hospital referral regions from the Dartmouth Atlas.²²

To create measures of exposure to low-value care for each PCP-attributed patient panel in the 20% samples, we started by identifying episodes of low-value care at the patient level. We adapted methods developed by Schwartz et al^16,23 for identifying low-value care in Medicare claims data and selected the 8 services that were most frequently used (Table 1). In addition, for each service, we defined and identified a denominator population among whom the service would be considered unnecessary (eAppendix 4 and eTable 2 in the Supplement). Although the services are often ordered by PCPs, our approach did not require that the PCP provided or ordered the low-value service; rather, we assessed low-value service exposure from all physicians of a PCP patient panel. The contribution of a PCP’s network of specialists to the provision of low-value care was compatible with several dimensions of care experiences examined in our analysis. For example, overall care ratings are not PCP specific, and overall PCP ratings may reflect patients’ valuation of specialty referrals made by their PCP.

We aggregated the beneficiary-level indicators of low-value service receipt into composite measures of exposure at the level of a PCP-attributed patient panel using a multivariate, multilevel model with the patients nested within the PCPs: logit Pr (LVS_Receipt_ijst = 1) = α_j + X_ijst′β_s and α_j ~ N(0, σ²), where LVS_Receipt_ijst is the binary indicator of whether patient i of PCP j received low-value service s in year t, and X_ijst is a vector of patient- and geography-related covariates (zip code–level covariates and hospital referral region) and year indicators. Exposure to low-value care for the patient panel of PCP j is represented by the random intercept α_j, which represents the relative propensity to receipt of low-value services among the patients in the PCP patient panel, after controlling for the vector of covariates in X_ijst, and accounts for sampling variation in the number of patients and potential services per PCP (eAppendix 5 in the Supplement).

We focused on 9 items in 2 sections of the CAHPS Medicare fee-for-service survey: “Your Health Care in the Last 6 Months” and “Your Personal Doctor,” defined as the “one you would see if you need a checkup, want advice about a health problem, or get sick or hurt” (Table 2). In the former section, we studied responses to questions about overall health care, waiting room times, and timely access to nonurgent and urgent care. In the latter, we assessed ratings of the personal physician overall and interactions with the personal physician. All of these items are commonly studied in the CAHPS literature^24,25 and could conceivably be associated with provision of low-value care.

All responses were linearly rescaled to a 0 to 10 scale for consistency in presentation of results (Table 2). We created patient-level composite scores for respondents’ interactions with their personal physician by averaging across standardized responses (ie, by first subtracting the grand item mean from individual responses), averaging across de-meaned responses, and adding back the mean of all grand item means. We did not expect bias from different nonresponse among these items because 98.8% of patients who responded to at least 1 of these items responded to all of the items.

Patient covariates in the model estimating low-value service exposure included age, sex, race/ethnicity, Medicaid-Medicare dual status in at least 1 month, and indicators for 27 chronic conditions. To further adjust for health status, we added an indicator of having 6 or more chronic conditions and also used the claims data to calculate Hierarchical Condition Category risk scores, which indicate higher future Medicare spending with higher scores. Zip code–level sociodemographic variables included median household income, percentage of patients in poverty, and percentage of patients with a college education.

For the main analysis of an association between CAHPS scores and low-value service exposure, we included the following patient variables from the CAHPS survey: age as a categorical variable (<65, 66-69, 70-74, 75-79, 80-84, and ≥85 years), Medicaid-Medicare dual eligibility status, highest level of education completed (less than high school, some high school, some college, college graduate, and above college), and self-reported overall physical health and overall mental or emotional health (poor, fair, good, very good, or excellent).

Statistical analysis was performed from January 1, 2019, to December 9, 2020. We categorized PCPs into deciles of the low-value service composite for their attributed patients, from least (decile 1) to most (decile 10) low-value care exposure. We first regressed CAHPS scores on decile indicators and patient covariates, with clustered standard errors to account for patient grouping within physicians, for a 9-df omnibus test of any differences across deciles. Our second analysis regressed CAHPS scores on the numerical decile index and covariates to test for a linear trend across deciles. To further describe trends, for each CAHPS model, we conducted statistical tests of the difference between the mean adjusted CAHPS score in each decile of the low-value service exposure composite and the overall adjusted mean CAHPS score. All statistical tests were 2-sided and conducted at an α level of .05.

Approximately 26 million beneficiary-years were used to create the low-value service composites for 100 743 PCP patient panels, with a mean of approximately 258 patients per PCP. Depending on the CAHPS item, there were between 135 657 and 330 600 respondents, ranging from 2 to 3.4 per PCP.

In outpatient facility claims for low-value services, the PCP was listed on 41.8% of services, as the attending physician (31.4%), operating physician (2.9%), or other physician (7.5%). Among professional claims for low-value services, the PCP was listed on 45.6% of services, either as the performing physician (10.0%) or referring physician (35.6%).

We observed substantial variation in low-value care exposure across patient panels, consistent with prior studies.²⁰ Figure 1 shows that the low-value service exposure composites are positively associated with use in each of the 8 services. We did not find meaningful differences between patient panels in the first and fifth quintiles of low-value service exposure (eAppendix 6 and eTables 3 and 4 in the Supplement).

For each CAHPS outcome, Figure 2 plots differences between the mean adjusted CAHPS score for each decile of the low-value service exposure measure and the overall mean adjusted CAHPS score across deciles. With the exception of waiting room time, trend coefficients were bounded to be very close to zero. The waiting room time item had a clear downward trend, with lower CAHPS scores (reporting more frequently late start of appointments) in the deciles of patients receiving the most low-value care. In particular, PCP patient panels who received the most low-value care responded with a CAHPS score that was a mean of 0.448 points lower than those who received the least low-value care, a difference 7.7 times larger than the maximum difference observed between any 2 deciles in any other outcome.

Table 3 summarizes the mean adjusted CAHPS scores across deciles of the low-value service exposure measure. With the exception of waiting room time, mean adjusted scores in each decile were above 8.5, with a maximum between-decile difference of 0.058, on a 10-point scale. A test of linear trend across the deciles found significant trends of CAHPS scores for 5 of 6 measures (compared with only 3 for the categorical specification). However, excluding waiting room time, the largest-magnitude coefficient indicated that a 1 higher decile of low-value service exposure would increase the CAHPS outcome by a mean of only 0.005 points, adjusting for other covariates. We also provide full regression tables (eAppendix 7 and eTables 5-8 in the Supplement), sensitivity analyses (eAppendices 8 and 9 and eTables 9 and 10 in the Supplement), and description of all code files (eAppendix 10 and eTables 11-15 in the Supplement).

We assessed the associations between PCP patient panels’ levels of exposure to low-value care and health care experience ratings. Across multiple items from CAHPS sections on “Your Health Care in the Last 6 Months” and “Your Personal Doctor,” we did not find that exposure to more low-value care was associated with better health care experience ratings. Although some associations were statistically significant, their magnitudes were substantially smaller than those typically considered meaningful in other CAHPS literature^26,27 and inconsistent in direction across levels of low-value service exposure. These findings challenge the claim that use of patient-reported experiences to measure quality of care may lead to unnecessary care and possibly iatrogenic injury and death.^6-9

The only notable association was that patient panels with higher exposure to low-value care reported more frequently having behind-schedule appointments. Specifically, patient panels with the highest low-value care exposure rated this item 0.448 points lower on a 10-point scale than those with the lowest low-value care exposure, a difference substantially larger than typically observed in comparisons of Medicare plans.²⁶ One possible interpretation is that poorly organized or overwhelmed practices substitute wasteful services for higher-value services that require more cognitive effort and clinician time.^28,29

Our study design, however, was correlational and did not support conclusions about causal mechanisms underlying any associations between low-value service provision and patient experience reports. It improves on the prior literature, which has been characterized by patient-level endogeneity from using the same individuals to measure both patient experiences and health outcomes or health care use.^7,18 For the research question our study addressed, such an analysis may potentially misattribute to receipt of low-value care what is actually an association of unobserved patient characteristics, such as preferences for amount and type of care; socioeconomic factors, such as educational level, income, and leisure time; and trust in their physician and the health care system more broadly. Our study design reduced this type of patient-level confounding by using independent samples, instead of the same sample, to assess low-value care provision and health care experiences. Furthermore, because the CAHPS sample was much smaller than the claims sample, there was little overlap between the 2 samples. In other words, patient-level confounding was necessarily limited as the characteristics of a CAHPS respondent were at most weakly associated with the mean characteristics of the PCP patient panel. The large claims sample also made possible a more precise estimation of low-value care exposure, and the variation in the estimated exposure supported informative comparisons of health care experiences between higher and lower levels of low-value care provision.

This study has limitations. First, we relied on claims data to identify diagnoses and procedures, which may not be as complete or accurate as medical records. Second, our analysis does not address systematic, unobserved differences in patient panels between physicians. For example, patients may sort to physicians based on the consonance between their preferences and a physician’s practice style. Third, although the CAHPS response rate was 41.9%, our interest was in the reports of patients who respond to such surveys. This response rate is typical of surveys of this type, and the respondents are likely representative of those who would respond to other patient surveys.

Fourth, to measure low-value service exposure, we counted all low-value services received by patients in a PCP patient panel, thereby attributing the provision of low-value care to the network of physicians who provided care to the PCP patient panel. Such PCP-centric models for identifying and establishing care networks are commonly used by researchers and policy makers.^30-35 Because only a share of low-value services were provided by a patient’s PCP, associations between low-value care provision and PCP-specific patient ratings may have been attenuated toward the null. However, we found similarly weak associations for CAHPS survey items associated with health care more broadly, including patients’ overall care rating, which may reflect care provided by all physicians involved in the patient’s care.

We did not find evidence that patients who are exposed to more low-value care rate their health care experiences better. They also generally do not rate their health care worse, which may be because they are unable to assess the value of care or because their physicians effectively correct information asymmetries through shared decision-making when denying low-value services. These findings are inconsistent with some highly cited works,^7,18 likely in part because our methodological approach is not subject to the same levels of patient-level confounding. Knowing this may help reduce the use of low-value care that is provided to appease patients who would be equally satisfied with less wasteful care and help alleviate concerns that patient dissatisfaction should inhibit waste reduction under alternative payment models.

Accepted for Publication: March 26, 2021.

Published Online: May 28, 2021. doi:10.1001/jamainternmed.2021.1974

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Sanghavi P et al. JAMA Internal Medicine.

Corresponding Author: Prachi Sanghavi, PhD, Biological Sciences Division, Department of Public Health Sciences, The University of Chicago, 5841 S Maryland Ave, MC2000, Chicago, IL 60637 (psanghavi@uchicago.edu).

Author Contributions: Dr Sanghavi had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Sanghavi, McWilliams, Zaslavsky.

Acquisition, analysis, or interpretation of data: Sanghavi, McWilliams, Schwartz.

Drafting of the manuscript: Sanghavi, Zaslavsky.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Sanghavi, McWilliams, Zaslavsky.

Obtained funding: Sanghavi, McWilliams, Zaslavsky.

Administrative, technical, or material support: Sanghavi, Schwartz.

Supervision: Sanghavi, McWilliams.

Conflict of Interest Disclosures: Dr McWilliams reported serving as an unpaid member of the board of directors for the Institute for Accountable Care and receiving personal fees from the Medicare Payment Advisory Commission for serving as a consultant on the application of low-value care measures, outside the submitted work. Dr Schwartz reported receiving personal fees from Lown Institute, CVS Health, and the Medicare Payment Advisory Commission outside the submitted work. No other disclosures were reported.

Funding/Support: This work was supported by a grant from the National Institute for Health Care Management Foundation and grant P01AG032952 from the National Institute on Aging.

Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Additional Contributions: Lin Ding, PhD, Harvard Medical School, provided programming support for this study. She was not explicitly compensated for this contribution.