Changes in Patient Experiences of Hospital Care During the COVID-19 Pandemic

Key Points

Question  Did patient hospital experiences change during the COVID-19 pandemic and, if so, did the changes differ across hospitals with different patient-staff ratios and prepandemic quality (hospital overall star rating)?

Findings  In this cohort study of 3 900 887 respondents of the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) discharged from 2020 to 2021, comparisons of HCAHPS scores for patients with prepandemic data indicated that all HCAHPS scores declined during 2020 to 2021, especially for hospitals with lower prepandemic staffing and performance. Declines were largest for staff responsiveness and cleanliness.

Meaning  The results of this study suggest that higher-staffed and higher-performing hospitals were more resilient to the conditions of the COVID-19 pandemic, but by late 2021, patients’ experience of care had declined in all hospitals.

Importance  It is important to assess how the COVID-19 pandemic was adversely associated with patients’ care experiences.

Objective  To describe differences in 2020 to 2021 patient experiences from what would have been expected from prepandemic (2018-2019) trends and assess correlates of changes across hospitals.

Design, Setting, and Participants  This cohort study compared 2020 to 2021 data with 2018 to 2019 data from 3 900 887 HCAHPS respondents discharged from 3381 HCAHPS-participating US hospitals. The data were analyzed from 2022 to 2023.

Main Outcomes and Measures  The primary outcome was an HCAHPS summary score (HCAHPS-SS), which averaged 10 HCAHPS measures. The primary analysis estimated whether HCAHPS scores from patients discharged from 2020 to 2021 differed from scores that would be expected based on quarterly and linear trends from 2018 to 2019 discharges. Secondary analyses stratified hospitals by prepandemic overall star ratings and staffing levels.

Results  Of the 3 900 887 HCAHPS 2020 to 2021 respondents, 59% were age 65 years or older, and 35% (11%) were in the surgical (maternity) service lines. Compared with trends expected based on prepandemic (2018-2019) data, HCAHPS-SS was 1.2 percentage points (pp) lower for quarter (Q) 2/2020 discharges and −1.9 to −2.0 pp for Q3/2020 to Q1/2021, which then declined to −3.6 pp by Q4/2021. The most affected measures (Q4/2021) were staff responsiveness (−5.6 pp) and cleanliness (−4.9 pp); the least affected were discharge information (−1.6 pp) and quietness (−1.8 pp). Overall rating and hospital recommendation measures initially exhibited smaller-than-average decreases, but then fell as much as the more specific experience measures by Q2/2021. Quietness did not decline until Q2/2021. The HCAHPS-SS fell most for hospitals with the lowest prepandemic staffing levels; hospitals with bottom-quartile staffing showed the largest decrements, whereas top-quartile hospitals showed smaller decrements in most quarters. Hospitals with better overall prepandemic quality showed consistently smaller HCAHPS-SS drops, with effects for 5-star hospitals about 25% smaller than for 1-star and 2-star hospitals.

Conclusions and Relevance  The results of this cohort study of HCAHPS-participating hospitals found that patient experience scores declined during 2020 to 2021. By Q4/2021, the HCAHPS-SS was 3.6 pp lower than would have been expected, a medium effect size. The most affected measures (staff responsiveness and cleanliness) showed large effect sizes, possibly reflecting high illness-associated hospital workforce absenteeism. Hospitals that were lower performing and less staffed prepandemic may have been less resilient to reduced staff availability and other pandemic-associated challenges. However, by Q4/2021, even prepandemic high-performing hospitals had similar declines.

Since 2008, Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey results from patients at short-term acute care hospitals have been publicly reported with quarterly updates on the US Centers for Medicare & Medicaid Services’ (CMS) Care Compare (formerly Hospital Compare) website and have been included in the Hospital Value-Based Purchasing Program payment formula since 2013.¹ Hospital CAHPS provides a consistent and detailed source of information for monitoring changes in patient experiences in response to changes in circumstances or policies. During the COVID-19 pandemic, hospitals suddenly faced extraordinary demands,² and before the availability of vaccines, health care professionals feared being exposed to COVID-19 at work and infecting family members. To comply with US Centers for Disease Control and Prevention guidelines associated with COVID-19 infection control, hospitals implemented new protocols involving social distancing, personal protective equipment (PPE) requirements of masks and face shields, and restricted visitor access, factors that were subsequently found to be associated with adverse patient outcomes.³ High rates of emotional exhaustion among health care workers have been attributed to staffing shortages, uncertainty around evolving hospital protocols, unreliable PPE supplies, and incivility directed toward health care workers.⁴ Limited information has suggested that emotional exhaustion increased greatly among staff, with nearly half of nurses in 1 annual survey reporting exhaustion during 2021 and 2022.⁴ At times, hospitals became overwhelmed as admissions spiked.⁵

Early evidence suggests that these COVID-19–related changes were negatively associated with health care quality. Enhanced isolation practices and PPE requirements are thought to have been associated with a reduced focus on routine infection control practices. After years of steady declines, increases in rates of health care–acquired infections were associated with COVID-19 surges.⁶ The demands that the COVID-19 pandemic imposed on hospitals and feelings of isolation associated with restrictive family visitation policies may have disadvantaged patients. Prepandemic research suggests that compared with nonisolated patients, patients who are isolated for infection control consistently exhibit more depression and anxiety, experience more adverse events,⁷ and report worse experiences of care on HCAHPS measures.⁸ Several other potential negative associations of the pandemic protocols, including substantial delays in services, challenges communicating with physicians and nurses, and poor patient education, were identified in a single-hospital study of patients with and without COVID-19.⁹

In this study, we used cross-sectional data from US hospitals to describe how HCAHPS scores during a peak period of the COVID-19 pandemic differed from what would have been expected based on prepandemic trends. We also assessed whether patients treated in hospitals with higher prepandemic staffing levels and a better quality of care prepandemic had smaller decreases in reported experience after the start of the pandemic. We expected that the pandemic would be associated with lower HCAHPS scores, especially in hospitals with lower prepandemic staffing levels and quality of care, because such hospitals would have fewer resources and processes in place to maintain positive patient experiences.

The analyses presented in this article used data for all quarters (Qs) from 2018 to 2021 from the 3381 hospitals that submitted to CMS at least 25 completed HCAHPS surveys from discharged patients during at least 1 quarter during 2020 and 2021 (eMethods 1 in Supplement 1). This cohort study was approved by the institutional review board at Rand Corporation; the requirement for written informed consent was waived because only deidentified data were obtained. We followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guidelines.¹⁰

The study’s primary outcome was the HCAHPS summary score (HCAHPS-SS), the average of 10 HCAHPS measures: 6 multi-item composite measures that were each given a weight of 1.0 (communication with nurses, communication with doctors, staff responsiveness, communication about medicines, care transition, and discharge information) and 4 single-item measures that were each given a weight of 0.5 (overall hospital rating, hospital recommendation, cleanliness, and quietness). Survey response options were “never,” “sometimes,” “usually,” or “always” for all measures other than discharge information (options were “yes” or “no”), care transition (options were “strongly disagree,” “disagree,” “agree,” or “strongly agree”), hospital recommendation (options were “definitely no,” “probably no,” “probably yes,” or “definitely yes”), and overall hospital rating (response options ranged from 0 [“worst possible hospital”] to 10 [“best possible hospital”]). Top-box scoring, which is the proportion of most-positive category responses (except for the overall hospital rating, for which 9 or 10 correspond to a top-box response), was used for all HCAHPS measures and the HCAHPS-SS. Secondary analyses examined each of the 10 constituent HCAHPS measures individually. The HCAHPS items and details of survey implementation protocols can be found at http://www.hcahpsonline.org.

For an indicator of overall prepandemic hospital quality, we used the 2019 hospital overall star rating from CMS,¹¹ which is based on 5 groups of measures associated with health care quality: mortality, readmissions, safety, patient experience (each group weighted 22%), and timely and effective care (weighted 12%). These star ratings were classified as 1 to 2 (pooled because of low frequency), 3, 4, 5, or missing.

Prior research has shown that nurse communication is the HCAHPS measure most strongly associated with overall HCAHPS ratings and recommendations^12,13 and that nurse staffing levels are strongly associated with HCAHPS scores.^12-16 Staffing level, from the 2019 American Hospital Association data, was defined as the number of full-time–equivalent nurses, registered nurses, and licensed practical nurses working in the hospital (excluding nursing home and long-term care unit staff) per 1000 adjusted patient days. We used a mean imputation for missing data (<1%) on nursing levels. Staffing levels were then classified into quartiles.

To make results comparable across hospitals and over time, HCAHPS data were adjusted for the mode of survey administration and patient characteristics not within the control of the hospital that were known to affect survey responses.^17,18 To evaluate the post-2019 departure of HCAHPS-SS from prepandemic trends, we estimated 8 adjusted linear mixed-effect regression models, each of which included 9 quarters of patient-level data. Each model included data from the 8 quarters that immediately preceded the pandemic (Q1/2018 to Q4/2019) to evaluate HCAHPS-SS in 1 quarter during the pandemic period (Q1/2020 to Q4/2021). Each linear mixed-effect regression model included patient-mix variables, an indicator of the mode of survey administration, hospital random effects, a linear time trend (number of quarters since Q1/2018), 3 quarter intercepts (indicators for each of Q2, Q3, and Q4, regardless of year) to control for seasonal trends, an indicator for hospitals voluntarily submitting HCAHPS data when reporting was optional (Q1/2020 and/or Q2/2020),¹⁹ and a post-2019 indicator (an indicator of being in a COVID-19–era quarter during 2020 or 2021). The last term estimated the departure of the post-2019 quarter in question from the season and linear trend. Each model included hospitals with 25 completed surveys during the specified post-2019 quarter and all data from that quarter and other quarters from 2018 to 2019 for which the hospital had 25 completed surveys. We repeated these analyses for each HCAHPS measure.

In addition to overall changes, we assessed whether hospital characteristics were associated with differential changes in HCAHPS scores by stratifying by hospital characteristics 1 at a time (2019 hospital overall star rating [1-2, 3, 4, 5, and no star] and quartiles of staffing levels) and graphically illustrating trends in changes in HCAHPS-SS by hospital characteristics. We also examined variation in estimated pandemic effects across the 9 US Census Divisions. We ran sensitivity analyses described in eMethods 3 in Supplement 1 to assess whether our use of varying cohorts of hospitals was associated with the study results and whether patterns in patient volume may have accounted for the observed trends. Analyses were conducted using SAS, version 9.4 (SAS Institute), and statistical significance was set at P < .05.

For the 3381 hospitals used in the primary analysis, HCAHPS response rates were 25%, 25%, 24%, and 22%, respectively, for 2018, 2019, 2020, and 2021,²⁰ resulting in 5 298 431 completed surveys during 2018 and 2019 and 3 900 887 completed surveys from 2020 to 2021. Of the 2020 to 2021 (2018-2019) respondents, 26% (26%) were in fair or poor self-rated health, 59% (57%) were 65 years or older, 35% (38%) were in the surgical service line, and 11% (11%) were in the maternity service line (eTables 1 and 2 in Supplement 1 show patient and hospital characteristics, respectively).

The first row of Table 1 and the dark blue dotted line in Figure 1 present the estimated differences of HCAHPS-SS scores from 2020 to 2021 from expected scores based on prepandemic trends. The HCAHPS-SS was not significantly different than expected based on prepandemic trends for Q1/2020 discharges but was 1.2 percentage points (pp) lower than expected (−1.2 pp) for Q2/2020 discharges and −1.9 to −2.0 pp for Q3/2020 to Q1/2021 and then steadily worse to −3.6 pp by Q4/2021. Figure 1 and eTable 3 in Supplement 1 show comparable analyses for specific HCAHPS measures; eFigure 1 in Supplement 1 shows simple HCAHPS-SS trends by quarter from Q1/2018 through Q4/2021.

By Q4/2021, 2 years into the COVID-19 pandemic, the most affected HCAHPS measures, with large²¹ departures from prepandemic trends, were staff responsiveness (−5.6 pp) and cleanliness (−4.9 pp), while communication about medicines (−4.2 pp), care transition (−3.9 pp), overall hospital rating (−3.9 pp), communication with nurses (−3.4 pp), hospital recommendation (−3.3 pp), and communication with doctors (−2.9 pp) were moderately²¹ affected. The least affected measures were discharge information (−1.6 pp) and quietness (−1.8 pp), neither of which exceeded a full top-box point less than what was expected until Q2/2021 or Q3/2021. The HCAHPS global measures, overall rating and hospital recommendation, had smaller-than-average initial declines but fell as much as the more specific measures by Q2/2021.

Rows 2 to 5 of Table 1 show the prepandemic linear trend (a small increase of approximately 1 pp per year) and the seasonal trends (scores were approximately 0.4 points higher April-September [Q2-Q3] than October-March [Q4-Q1]). Accounting for these trends allowed for a more accurate estimation of the departures of 2020 to 2021 scores from prepandemic trends.

As shown in Table 2 and Figure 2, HCAHPS-SS fell the most for hospitals with the lowest prepandemic staffing levels: hospitals with bottom-quartile (lowest) staffing experienced the largest decrements, while top-quartile (highest) hospitals had smaller decrements during most quarters. However, by Q4/2021, the hospitals with the highest prepandemic staffing had similar decrements as the hospitals with the lowest prepandemic staffing (with a range −3.5 to −3.7 pp across groups).

Hospitals with better overall quality (as indicated by hospital overall star rating) before the pandemic showed consistently smaller declines in HCAHPS-SS during the pandemic, with effects for 5-star hospitals about 25% smaller than those for 1-star and 2-star hospitals in most quarters (Table 2; Figure 3). However, by Q3/2021, hospitals in all star rating groups had dropped a full pp, narrowing the difference so that by Q4/2021, the effect for 5-star hospitals was only 13% smaller than that for 1-star and 2-star hospitals (−3.2% for 5 stars vs −3.7% for 1 to 2 stars).

Results were similar when restricting to smaller, constant cohorts of hospitals (eFigures 2-4 in Supplement 1). Patient volume changes across the interval were limited except for a sharp decrease in Q2/2022 that was followed by a recovery in Q3/2022, which did not correspond to trends in the HCAHPS-SS (eFigure 5 in Supplement 1).

As shown in eTables 4A and B and eFigure 6 in Supplement 1, HCAHPS scores decreased in all US Census divisions. There was little regional variation.

In this cohort study of US hospitals, patient hospital experiences as measured by the HCAHPS survey were worse than expected based on prepandemic trends during the COVID-19 pandemic. This unprecedented decline in HCAHPS scores was seen in every region of the US, with little regional variation. Hospitals with higher staffing levels and better overall prepandemic quality were more resilient and slower to decline, but eventually they also declined.

By Q4/2021, a summary measure of patient experiences was 3.6 pp lower across all hospitals than would have been expected without the pandemic, a medium effect size by the guidelines by Quigley et al²¹ for patient experience measures. Staff responsiveness and cleanliness scores decreased by large amounts, possibly reflecting high absenteeism in the hospital workforce and delays in care associated with staff having to wear PPE.^9,22 The decline of the overall rating and hospital recommendation measures lagged the decline in more specific HCAHPS measures, possibly reflecting initial forbearance or lower expectations in patients’ overall assessments at the onset of the pandemic. However, as the pandemic continued, the global ratings also declined to a similar extent as the more specific measures of patient experience. This pattern suggests that global ratings alone do not fully capture changes in specific patient experiences,²³ nor do they identify the areas in need of improvement, especially in the face of unusual events, such as a pandemic. Deficits were largest for hospitals that during the prepandemic period were lower performing and had lower staffing levels. These hospitals may have been less resilient to reductions in staff availability during the COVID-19 pandemic and less able to adapt to unprecedented challenges in the health care environment. However, 2 years into the pandemic, the initially resilient hospitals’ HCAHPS scores had declined almost as much as those of the prepandemic lower-performing hospitals. The emergence of the highly contagious Omicron variant, ongoing staffing shortages, and cumulative effects of staff burnout may have compromised patient experiences of care in all hospitals.^24,25

This study has several potential limitations. First, response rates were modest, and nonresponse bias may have affected the findings. However, research on Consumer Assessment of Healthcare Providers and Systems surveys has found little evidence of nonresponse bias after adjustment for patient mix.¹⁸ Response rates declined slightly throughout 2018 to 2021, which were consistent with prepandemic trends and trends for surveys in general.²⁶ Second, the study used observational data; the analyses and conclusions were limited to the description of the departures of 2020 to 2021 HCAHPS scores from 2018 to 2019 trends. Unobserved factors may be responsible for the changes, as is the case for all observational data. Third, we used linear rather than logistic regression models to assess trends in top-box scores because linear models are easier to interpret and are almost identical when sample sizes are large and outcomes are predominantly between 20% and 80%.^18,27 Fourth, the hospital overall star rating measure used to stratify hospitals included prepandemic hospital patient experience scores, with a 22% weight. However, there was a positive association between HCAHPS and the other components of the overall star rating. A modified version of the overall star ratings that removed the HCAHPS star ratings had a correlation of 0.985 with the hospital overall star rating (results not shown). As such, these results were not sensitive to the inclusion of HCAHPS in the prepandemic quality scores. Further, although one would expect some regression to the mean, which would be associated with slightly larger decreases for hospitals with higher prepandemic HCAHPS scores, such effects were accounted for in the regression models. Finally, prepandemic HCAHPS scores were based on potentially different, possibly less severe cases than post-2019 HCAHPS scores, which may have affected patient ratings. For example, in the Medicare Consumer Assessment of Healthcare Providers and Systems survey, patients with end-stage kidney disease and diabetes tend to give more positive ratings than otherwise similar patients.²⁸ Nonetheless, self-rated health of HCAHPS patients was similar from 2018 to 2019 and 2020 to 2021, and scores were adjusted for self-rated health status and service lines. There were also strengths to this study. For example, the large volume and long tenure of HCAHPS data and their specific measures make it possible to understand which aspects of inpatient care were most affected by the pandemic, their course over time, and how resilience in the face of the COVID-19 pandemic varied regionally and by hospital characteristics.

The results of this cohort study found that HCAHPS patient experience scores declined from 2020 to 2021. The most affected measures (staff responsiveness and cleanliness) showed large effect sizes, possibly reflecting high illness-associated hospital workforce absenteeism. Hospitals that were higher performing and higher staffed prepandemic were slower to decline, but by Q4/2021, scores for even these hospitals had declined.

Accepted for Publication: June 29, 2023.

Published: August 25, 2023. doi:10.1001/jamahealthforum.2023.2766

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2023 Elliott MN et al. JAMA Health Forum.

Corresponding Author: Marc N. Elliott, PhD, RAND Corporation, 1776 Main St, Santa Monica, CA 90401 (elliott@rand.org).

Author Contributions: Mr Cohea had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Elliott, Lehrman, Giordano, Goldstein.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Elliott, Beckett, Goldstein.

Critical review of the manuscript for important intellectual content: Elliott, Beckett, Cohea, Lehrman, Cleary, Giordano, Russ, Fleisher.

Statistical analysis: Elliott, Cohea, Russ, Goldstein.

Obtained funding: Giordano.

Administrative, technical, or material support: Lehrman, Giordano, Goldstein.

Supervision: Giordano.

Conflict of Interest Disclosures: Dr Elliott reported grants from the US Centers for Medicare & Medicaid Services (CMS) during the conduct of the study. Drs Lehrman, Goldstein, and Fleisher reported being employees of CMS. Mr Cohea reported being a contractor for the Health Services Advisory Group during the conduct of the study. Dr Cleary reported grants from Yale School of Public Health during the conduct of the study. Mss Giordano and Russ reported holding a contract with CMS during the conduct of the study. No other disclosures were reported.

Funding/Support: The research was funded by CMS under contract GS-10F-0166R/HHSM-500-2017-00077G to the Health Services Advisory Group.

Role of the Funder/Sponsor: There were authors from the sponsoring organization (CMS), who, like other authors, participated in the design and conduct of the study, the analysis and interpretation of the data, the preparation and review of the manuscript, and the decision to submit the manuscript for publication.

Data Sharing Statement: See Supplement 2.

Additional Contributions: We thank Biayna Darabidian, MPP, and Katherine Osby, BA, RAND Corporation, for assistance with preparation of the manuscript. These individuals were not compensated beyond their salaries.