Effectiveness of an Analytics-Based Intervention for Reducing Sleep Interruption in Hospitalized Patients: A Randomized Clinical Trial

Key Points

Question  Can a clinical decision support tool, powered by a real-time prediction algorithm, help reduce delirium incidence and identify hospitalized patients who can safely forgo nighttime vital sign checks?

Findings  In this randomized clinical trial of 1930 inpatient encounters in 1699 patients, no difference was found between groups in delirium incidence, but physicians usually agreed with the assessment of the clinical decision support tool and therefore discontinued overnight vital sign checks. The intervention group experienced 31% fewer vital sign checks per night with no change in the rates of intensive care unit transfer or code blue alarms.

Meaning  The results of this trial indicate that augmenting physician judgment with a real-time prediction algorithm can help provide patients greater sleep opportunity without an increased risk of clinical decompensation.

Importance  Sleep has major consequences for physical and emotional well-being. Hospitalized patients experience frequent iatrogenic sleep interruptions and there is evidence that such interruptions can be safely reduced.

Objective  To determine whether a clinical decision support tool, powered by real-time patient data and a trained prediction algorithm, can help physicians identify clinically stable patients and safely discontinue their overnight vital sign checks.

Design, Setting, and Participants  A randomized clinical trial, with inpatient encounters randomized 1:1 to intervention vs usual care, was conducted from March 11 to November 24, 2019. Participants included physicians serving on the primary team of 1699 patients on the general medical service (not in the intensive care unit) of a tertiary care academic medical center.

Interventions  A clinical decision support notification informed the physician if the patient had a high likelihood of nighttime vital signs within the reference ranges based on a logistic regression model that used real-time patient data as input. The notification provided the physician an opportunity to discontinue measure of nighttime vital signs, dismiss the notification for 1 hour, or dismiss the notification for that day.

Main Outcomes and Measures  The primary outcome was delirium, as determined by bedside nurse assessment of Nursing Delirium Screening Scale scores, a standardized delirium screening tool (delirium diagnosed with score ≥2). Secondary outcomes included mean nighttime vital sign checks. Potential harms included intensive care unit transfers and code blue alarms. All analyses were conducted on the basis of intention-to-treat.

Results  A total of 1930 inpatient encounters in 1699 patients (intervention encounters: 566 of 966 [59%] men; mean [SD] age, 53 [15] years) were randomized. In the intervention vs control arm, there was a significant decrease in the mean (SD) number of nighttime vital sign checks (0.97 [0.95] vs 1.41 [0.86]; P < .001) with no increase in intensive care unit transfers (49 [5%] vs 47 [5%]; P = .92) or code blue alarms (2 [0.2%] vs 9 [0.9%]; P = .07). The incidence of delirium was not significantly reduced (108 [11%] vs 123 [13%]; P = .32).

Conclusions and Relevance  While this randomized clinical trial found no difference between groups in the primary outcome, delirium incidence, the secondary findings indicate that a real-time prediction algorithm embedded within a clinical decision support tool in the electronic health record can help physicians identify clinically stable patients who can forgo routine vital sign checks, safely giving them greater opportunity to sleep. Other aspects of hospital care that depend on clinical stability, such as level of care or cardiac monitoring, may be amenable to a similar intervention.

Trial Registration  ClinicalTrials.gov Identifier: NCT04046458

Insomnia in hospitalized adults is a common condition and a source of patient dissatisfaction.^1,2 Even short-term sleep loss impairs short-term memory and learning, which is a source of concern for inpatients who are expected to consent to major procedures and learn to use new medications or devices.³ Short-term sleep loss has been shown to decrease insulin sensitivity and increase blood pressure.^4-6 Use of earplugs and other nonpharmacologic sleep aids in the intensive care unit (ICU) suggests that improved sleep may be associated with reductions in delirium.⁷

Iatrogenic interruptions are a major cause of sleep deprivation in hospitalized patients.^1,8-10 When asked about interruptions that disturb sleep, patients cite nighttime vital sign checks, which are traditionally done every 4 hours despite a lack of evidence for the utility of this tradition.^10-13 Studies suggest that nighttime vital sign checks could be safely reduced in low-risk medical inpatients,^14,15 and one study used clinical decision support (CDS) to encourage physicians to reduce such checks.¹⁵ Without a randomized control group, however, the effect of the sleep promotion CDS could not be determined. In addition, the decision support did not include an assessment of order appropriateness. Whether CDS, including analytics to target sleep promotion interventions to appropriate patients, would be safe and effective is unclear.

We developed a predictive algorithm to identify patients who are at low risk for abnormal nighttime vital signs and deployed this model in a real-time CDS tool to assist hospitalists in identifying low-risk patients and ordering for those patients a reduction in vital sign check frequency to 3 times a day while awake (ie, sleep promotion vitals [SPV]). We evaluated this CDS tool in a randomized clinical trial embedded within the electronic health record (EHR) on a large medical service.

To train an algorithm to predict the outcome of nighttime vital signs within the reference ranges (referred to hereafter as normal), we assembled a data set of 5195 inpatient encounter-nights from the general medicine service at the University of California, San Francisco (UCSF), Medical Center during the latter half of 2016. This study was reviewed and approved by the institutional review board of the University of California, San Francisco. The need for informed consent was waived because the research is no more than minimal risk to participants. This study followed the Consolidated Standards of Reporting Trials (CONSORT) reporting guideline.

Because hospitalizations are dynamic and often short, each encounter-night was considered a separate outcome for the prediction algorithm. The predictors included in the training data were chosen based on face validity: patient age, hospital day, daytime vital signs, daytime laboratory test values, and prior night vital signs. Self-reported race and ethnicity were not considered factors in the risk of normal or abnormal vital signs on a given night and, while collected as demographic characteristics, data on race and ethnicity were not obtained.

To mimic how physicians gauge the instability of patients, each vital sign was categorized into clinically relevant categories (eg, heart rate, 100-119 bpm = 1, 120-139 bpm = 2, and ≥140 bpm = 3) and this was done separately for high and low systolic blood pressure, diastolic blood pressure, heart rate, and respiratory rate (eTable 1 in the Supplement). These scores were then summed during each patient daytime period to yield a variable that would suggest, for example, that a patient's systolic blood pressure was very abnormal on multiple checks.

A logistic regression model was chosen for the prediction task because of its interpretability and the simplicity of its implementation in the EHR (eTable 2 in the Supplement). The model was trained on 70% of the data set and its performance was measured on the remaining 30%. The model was able to correctly predict normal nighttime vital signs 84% of the time (positive predictive value) and abnormal vital signs 70% of the time (negative predictive value).

To implement the model for real-time use within our EHR system (EPIC, version 2019), we expressed the logistic model in the standard predictive model markup language, mapped the variables used by our model from the reporting database (Clarity) to equivalent versions in the live EHR database (Chronicles), and uploaded it into the Epic Cognitive Computing Platform for triggered implementation of the algorithm and integrated display of the results. To avoid alarm fatigue, we created a cutoff probability for the model's output that would trigger notification. The threshold of 0.9 (90% predicted likelihood of normal vital signs overnight) was chosen as a balance between safety (positive predictive value of 94%) and potential effect (desire to reduce sleep interruptions in a large proportion of hospitalized patients).

We designed and built an SPV CDS alert (eFigure 1 in the Supplement) that used our EHR sidebar feature, which is revealed from the right-hand side of the screen without obstructing the user from continuing their current tasks, consistent with the nonemergent nature of the information. The alert notifies the user that the patient is very likely (predicted probability >0.9) to have normal vital signs during the upcoming night, includes details about variables that contributed to the prediction, and gives the user 3 options: order SPV, delay the notification for 1 hour, or delay the notification until the next day.

To target our CDS tool effectively, we limited our notification to appear only for primary team members (eg, attending, resident) during daytime hours, and only for patients on general medicine services who were outside of the ICU.¹⁶

To determine how often the CDS notification would display and assess the user’s interaction with it, we piloted the notification on the direct-care (attending only) general medicine service at UCSF Medical Center from November 2 to December 20, 2018, and asked for informal, qualitative feedback. The pilot testing demonstrated that the notification frequency was tolerable to the users and that they often responded to it by ordering SPV as intended.

We designed a parallel randomized clinical trial to estimate the effects of the intervention on the general medicine service at UCSF Medical Center. The EHR software randomized patient encounters (1:1 ratio) to the SPV intervention vs a usual care control (Figure). All users opening the record of a patient who met the predicted probability threshold and other CDS targeting criteria described above and was randomized to the SPV intervention were exposed to the SPV sidebar alert.

The primary outcome of the trial was delirium, defined by a Nursing Delirium Screening Scale (Nu-DESC) score greater than or equal to 2.¹⁷ Nurses record a Nu-DESC score once per shift. An encounter with a Nu-DESC score greater than or equal to 2 was marked as a patient experiencing the outcome of delirium. Data for all outcomes of this trial were obtained from the reporting database of the EHR at the UCSF Medical Center.

Our secondary outcomes were sleep opportunity and patient satisfaction. The former is a solely EHR-based sleep metric developed at our institution, which was being used for a quality improvement campaign. The metric is calculated between 11 pm and 6 am and reflects the maximum time between iatrogenic interruptions (ie, blood pressure checks, non–as-needed medication administrations, and fingerstick glucose level checks) on a given encounter-night. This metric was chosen because of its face validity: if we reduce nighttime vital sign checks, which are a source of iatrogenic interruptions, we expect to see higher sleep opportunity. This metric is currently being validated against wrist actigraphy at our institution. Patient satisfaction was determined using the nationally standardized Hospital Consumer Assessment of Healthcare Providers and Systems survey about their stay. Although no question in the survey directly addresses sleep, the question, “How often was the area around your room quiet at night?” was believed to be the closest proxy and the patient’s response was included as one of our outcomes. We also assessed several process measures, including encounters with an SPV order and encounter-level mean number of blood pressure checks as a marker for vital sign checks.

We measured a series of balancing outcomes to ensure that reduced vital sign checks were not causing harm: code blue alarm, rapid response calls, and ICU transfer. We reported these outcomes every quarter to an interim safety monitoring board that evaluated the outcomes based on predetermined stopping criteria for the trial.

The trial implementation was as follows: (1) at 3 pm every day, the algorithm was run on EHR data for all non-ICU patients on the hospital medicine service who were not previously randomized, (2) patients whose predicted probability exceeded the threshold were then block-randomized to either the intervention or control group, and (3) the intervention arm patients had the CDS notification displayed to the user and the control arm patients had nothing displayed to the user. The encounter number, predicted probability, time stamp of notification, and user’s response to notification were all recorded in a custom table of the EHR database for later analysis.

A sample size calculation before initiation of the trial used the mean (0.75) and SD (1.52) of Nu-DESC scores on 3 general medicine units at UCSF Medical Center in the previous year, including our acute care for elderly patients unit. With an estimated effect size of 20% and a power level of 80%, we estimated we would require a sample size of 1612 patients in each trial arm. With 660 discharges per month from these units, a study period of 5 months would be required.

Study outcomes were analyzed across the trial arms using t tests (or Mann-Whitney tests if nonnormal) for continuous measures and χ² tests for categorical measures. All analyses were conducted on the basis of intention-to-treat. Testing for significance was unpaired and 2-sided and used a threshold α level of 0.5%. All statistical analyses were performed using Stata, version 16 (StataCorp LLC), and R, version 3.4.4 (R Foundation for Statistical Computing).

A total of 3025 encounters were screened and a total of 1930 encounters were randomized from March 11 to November 24, 2019: 966 to physician notification and 964 without notification (Figure). The baseline characteristics of the patients are presented in Table 1; demographic factors were similar between the 2 trial arms. Of the 966 encounters in the intervention arm, 400 were in women (41%) and 566 were in men (59%); mean (SD) age was 53 (15) years.

The notification provided physicians with the ability to order SPV, and physicians did so from within the notification window 60% of the time (eTable 3 in the Supplement). As a result, a near doubling of encounters occurred in the intervention arm with this order (770 [80%] vs 430 [45%]; P < .001) (Table 2). Subsequently, there was a 31% reduction in encounter-level mean (SD) blood pressure checks per night (0.97 [0.95] vs 1.41 [0.86]; P < .001). In 35% of the encounter-nights with an active SPV order, 1 or more vital sign checks were nonetheless performed.

The outcome of delirium, as measured by encounters with a nurse-reported Nu-DESC score of 2 or greater, was not significantly different between the intervention and usual care arms (108 [11%] vs 123 [13%]; P = .32). The outcome of sleep opportunity demonstrated a statistically significant difference between the 2 arms (mean [SD], 4.95 [1.45] vs 4.57 [1.30] hours; P < .001). Postdischarge Hospital Consumer Assessment of Healthcare Providers and Systems surveys, which ask patients about noise in or around their room at night, were completed by only 5% of the patients (intervention, 53; control, 49) and revealed no significant difference for this issue (P = .86). The set of balancing or safety outcomes assessed in this trial (code blue alarms, rapid response calls, ICU transfers, and death before discharge) demonstrated no significant difference between the study arms.

The results of this randomized clinical trial suggest that a predictive algorithm, paired with targeted and informative CDS, can assist physicians in reducing overnight vital sign checks that disrupt sleep for hospitalized patients, with no adverse effects on measures of patient safety. The intervention did not significantly reduce the incidence of delirium. Our study used a sleep-promotion intervention that was software based, and thus is more sustainable and scalable than interventions that require quality improvement staff to perform frequent educational campaigns or audit and feedback.

This study builds on the work of Arora et al¹⁵ by demonstrating that the appropriateness of an SPV order can be screened by a real-time predictive algorithm before submitting the recommendation to the physician for consideration. This process substantially reduces the cognitive burden on physicians of having to evaluate every patient on their list every day to determine who is clinically stable. This study also supports the literature on EHR-based randomized clinical trials, which can reduce the cost and effort of a trial because randomization, intervention deployment, and data collection can occur in software, without the involvement of a research assistant.¹⁸

Although our intervention resulted in a substantial increase in SPV orders, there was a smaller decrease in vital sign checks and no significant decrease in the incidence of delirium. In attempting to understand the diminishing effect sizes as we move from process measures, such as SPV orders, to outcome measures, such as delirium, it is useful to draw the causal chain of events in which our trial operated (eFigure 2 in the Supplement). When the prediction algorithm alerted the physician to the potential benefit of an SPV order, it is reasonable that the physician may have disagreed with the recommendation and thus broken the causal chain. Our data revealed that physicians did not order SPV 40% of the time (to minimize alert fatigue we did not require a reason). Next, although the physician may have enacted the SPV order, the order must be carried out by the bedside patient-care assistant or nurse, but it was not on 35% of the encounter-nights, which suggests that busy patient-care assistants and nurses may check vital signs out of habit without noticing that the order has changed for some of the patients. In addition, vital sign checks are only one of the sources of sleep disruption for hospitalized patients. Our intervention does not directly target factors such as room cleaning or noise from a neighboring patient, which could reasonably overwhelm the effect of our intervention. Notably, although recruiting more patients may have resulted in a statistically significant difference in the incidence of delirium, the small effect size of the study on this outcome (15 fewer instances of delirium over an 8-month trial) is not viewed differently based on the power calculation, which uses a purely estimated effect size.

Given the contextual factors described above, a method for strengthening the effect size could include targeting the intervention to patients at high risk of delirium, such as those on the acute care for elderly patients unit, and giving patient-care assistants a visual cue near the door for patients on SPV so they do not enter the room during late-night vital sign rounds.

This study has limitations. First, our predictive algorithm was developed using only data from our institution. Sites interested in implementing a similar intervention would likely need to refit the predictive algorithm on their data before using it for the intervention. However, the variables included in the model are common data from inpatient care, such as vital signs and routine blood tests, which makes our model less likely to be encoding latent, institution-specific information. Second, our effect size may be falsely lower than it could have been if SPV orders were carried out with higher fidelity by frontline staff. Third, our trial used encounter-level randomization, which could result in a falsely low effect size if physicians who saw the notification on patients in the intervention arm were motivated to reduce vital sign checks on patients in the nonintervention arm (ie, bleed over).

Results of this randomized clinical trial suggest that the model we have described herein, a predictive algorithm that identifies clinically stable patients for whom a hospital intervention can be safely discontinued, has potential applications beyond measurement of vital signs. Continuous cardiac monitoring, higher level of care (ie, stepdown unit), and routine daily blood tests are all scenarios that could benefit from a similar approach.

Accepted for Publication: September 21, 2021.

Published Online: December 28, 2021. doi:10.1001/jamainternmed.2021.7387

Corresponding Author: Nader Najafi, MD, University of California, San Francisco, 521 Parnassus Ave, Room 104, Box 0131, San Francisco, CA 94143 (nader.najafi@ucsf.edu).

Author Contributions: Dr Najafi had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Najafi, Robinson, Patel.

Acquisition, analysis, or interpretation of data: Najafi, Robinson, Pletcher.

Drafting of the manuscript: Najafi, Patel.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Najafi, Pletcher.

Obtained funding: Najafi, Patel.

Administrative, technical, or material support: Robinson, Pletcher, Patel.

Supervision: Patel.

Conflict of Interest Disclosures: None reported.

Funding/Support: This study was funded by the Learning Health System program of the Clinical and Translational Science Institute at the University of California, San Francisco.

Role of the Funder/Sponsor: The University of California, San Francisco, had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Data Sharing Statement: See Supplement 3.

Additional Contributions: Andrew Auerbach, MD, MPH, Ralph Gonzales, MD, MSPH, Kathy Lanier, BS, and the rest of the Learning Healthcare System Oversight Committee provided support, advice, and encouragement. No financial compensation was provided.