Flow diagrams outlining the selection process and generation of the final analytical cohorts. ICU indicates intensive care unit.
Colors and pairs correspond to individual ICU-level data provided in eTable 2 in the Supplement.
eTable 1. Staffing Patterns of Intervention ICUs
eTable 2. Characteristics of the 8 pairs of Intervention and Matched Control ICUs
eTable 3. Risk-Adjusted Mortality and LOS for Each Intervention and Control ICU During the Pre- and Post- Periods
eTable 4. Unadjusted Outcomes in Intervention and Control ICUs Stratified by Patients’ Predicted ICU Mortality (Low, Intermediate, and High)
eTable 5. Unadjusted Mortality and LOS for Patients in Larger, More Technologically Advanced Telemedicine ICUs and Control ICUs (VA ICU levels 1 and 2)
eTable 6. Risk-Adjusted Odds of Mortality and Relative Length of Stay (RLOS) for Patients in Larger, More Technologically Advanced Telemedicine ICUs and Control ICUs (VA ICU Levels 1 and 2)
eTable 7. Unadjusted Mortality and LOS for Patients in Smaller, Less Technologically Advanced Telemedicine ICUs and Control ICUs (VA ICU Levels 3 and 4)
eTable 8. Risk-Adjusted Odds of Mortality and Relative Length of Stay (RLOS) for Patients in Smaller, Less Technologically Advanced Telemedicine ICUs and Control ICUs (VA ICU Levels 3 and 4)
eTable 9. Power analyses for absolute mortality reduction in ICU and hospital mortality
eAppendix. Statistical Analysis
Nassar BS, Vaughan-Sarrazin MS, Jiang L, Reisinger HS, Bonello R, Cram P. Impact of an Intensive Care Unit Telemedicine Program on Patient Outcomes in an Integrated Health Care System. JAMA Intern Med. 2014;174(7):1160-1167. doi:10.1001/jamainternmed.2014.1503
Intensive care unit (ICU) telemedicine (TM) programs have been promoted as improving access to intensive care specialists and ultimately improving patient outcomes, but data on effectiveness are limited and conflicting.
To examine the impact of ICU TM on mortality rates and length of stay (LOS) in an integrated health care system.
Design, Setting, and Participants
Observational pre-post study of patients treated in 8 “intervention” ICUs (7 hospitals within the US Department of Veterans Affairs health care system) during 2011-2012 that implemented TM monitoring during the post-TM period as well as patients treated in concurrent control ICUs that did not implement an ICU TM program.
Implementation of ICU TM monitoring.
Main Outcomes and Measures
Unadjusted and risk-adjusted ICU, in-hospital, and 30-day mortality rates and ICU and hospital LOS for patients who did or did not receive treatment in ICUs equipped with TM monitoring.
Our study included 3355 patients treated in our intervention ICUs (1708 in the pre-TM period and 1647 in the post-TM period) and 3584 treated in the control ICUs during the same period. Patient demographics and comorbid illnesses were similar in the intervention and control ICUs during the pre-TM and post-TM periods; however, predicted ICU mortality rates were modestly lower for admissions to the intervention ICUs compared with control ICUs in both the pre-TM (3.0% vs 3.6%; P = .02) and post-TM (2.8% vs 3.5%; P < .001) periods. Implementation of ICU TM was not associated with a significant decline in ICU, in-hospital, or 30-day mortality rates or LOS in unadjusted or adjusted analyses. For example, unadjusted ICU mortality in the pre-TM vs post-TM periods were 2.9% vs 2.8% (P = .89) for the intervention ICUs and 4.0% vs 3.4% (P = .31) for the control ICUs. Unadjusted 30-day mortality during the pre-TM vs post-TM periods were 7.7% vs 7.8% (P = .91) for the intervention ICUs and 12.0% vs 10.2% (P = .08) for the control ICUs. Evaluation of interaction terms comparing the magnitude of mortality rate change during the pre-TM and post-TM periods in the intervention and control ICUs failed to demonstrate a significant reduction in mortality rates or LOS.
Conclusions and Relevance
We found no evidence that the implementation of ICU TM significantly reduced mortality rates or LOS.
Intensive care unit (ICU) telemedicine (TM) programs have been heralded as a potential solution to a number of vexing problems facing critical care, including a finite supply of intensivists, difficulty in attracting and retaining intensivists to smaller rural hospitals, and reductions in the availability of trainees in academic medical centers.1- 3 Despite great promise, empirical data evaluating the impact of ICU TM programs are limited.4
As with most multifaceted system-level interventions, evaluating the impact of ICU TM programs is complex. Programs often involve several related interventions, including an electronic health record, standardization of treatment protocols, changes in ICU staffing, and enhanced patient monitoring.1,5 Although ICU TM has the potential to affect numerous intermediate outcomes (eg, nosocomial infections, ventilator days), a primary motivation has always been the potential to reduce mortality rates.
Early single-center studies by inventors of the ICU TM technology suggested that these programs significantly reduced mortality rates.1,6 More recent studies have provided a more complex picture.7- 9 Limitations of prior studies include a lack of concurrent control groups and the inability to look at longer-term outcomes (eg, 30-day mortality rates). Furthermore, most prior studies were conducted within academic health care systems, and few included smaller rural facilities that might stand to benefit the most from improved access to skilled intensivists.
To better understand the impact of ICU TM, we used a quasi-experimental design to evaluate the impact of TM implementation on short-term (ICU and in-hospital) and longer-term (30-day) mortality rates and length of stay (LOS) within a regional network of 7 hospitals in the US Department of Veterans Affairs (VA).
Our study was approved by the institutional review board at the University of Iowa and Iowa City VA Health Care System. No patient consent was deemed necessary by both boards.
Our study was conducted in a network of 7 VA hospitals (8 ICUs; 73 beds) located within a regional VA health care network in the upper Midwest. All hospitals used the same electronic health record; this record includes clinic notes, computerized order entry, and patient laboratory and radiographic test results.
In 2011, the network began implementing a state-of-the-art ICU TM system. This system included (1) a central telemonitoring center located at the Minneapolis VA hospital, staffed 21 hours per day, 7 days per week with an experienced intensivist plus 2 critical care nurses; (2) a clinical information system that provided the monitoring center with real-time access to patient vital signs, intravenous infusion rates, and ventilator settings; and (3) high-speed video and audio connections between all ICU patient rooms and the monitoring center. The systems provided alerts to the TM staff when laboratory values were abnormal or vital signs exceeded prespecified parameters. The TM system was implemented in a staggered manner across the 7 hospitals between August 2011 and February 2012. The ICU staffing patterns and use of standardized order sets were not altered by the introduction of the ICU TM program.
Our study included 3 academic medical centers, 1 small urban hospital, and 3 rural hospitals. The 8 ICUs included 1 medical ICU, 1 surgical ICU, and 6 mixed ICUs (see eTable 1 in the Supplement for further details regarding the ICUs). In 6 of the ICUs, the TM staff were authorized to monitor patients and make interventions as they deemed appropriate. For the 2 remaining ICUs, the monitoring center staff could intervene only when explicitly requested. Nurses and physicians in the monitoring center evaluated new admissions and patients whose condition was deteriorating and then contacted the bedside provider with recommendations. In emergency situations (eg, cardiac arrest or hemodynamic instability), the monitoring staff were preauthorized to intervene at all sites.10
Our analysis included both a pre-post comparison (with each ICU that received the TM system serving as its own control) and comparison with concurrent control groups (ie, VA ICUs that did not receive the ICU TM system). Specifically, we matched each intervention ICU with a single control ICU drawn from the 121 VA ICUs that were not part of the ICU TM program. Intervention and control ICUs were matched according to (1) ICU type (medical, surgical, or mixed); (2) ICU admission volume; and (3) racial mix, measured as percentage of ICU admissions with patients categorized as white. After identifying a list of potential control ICUs for each ICU receiving the TM system, we reviewed geographic location and VA ICU level (ICU levels scored as 1-4, with level 4 providing the most basic ICU services [eg, telemetry monitoring] and level 1 providing comprehensive tertiary care) to select a final control ICU (eTable 2 in the Supplement).
After completing the matching process, we identified all admissions to each intervention ICU during the 6 months before TM implementation (pre-TM period) and the 6 months after TM implementation (post-TM period). For each control ICU, admissions were identified using the same pre-TM and post-TM time windows as their respective intervention ICU. We obtained patient-level data from an array of VA administrative files that have been used previously in evaluating ICU outcomes.11,12 These files contained (1) patient demographics; (2) primary diagnosis and comorbid illnesses, as captured in International Classification of Diseases, 9th Revision, Clinical Modification codes; (3) patient location (ie, ICU or floor); (4) laboratory values; and (5) mortality rates in the ICU, in the hospital, or after discharge.
After identifying all admissions to our intervention and control ICUs, we applied several exclusion criteria. First, we limited our cohort to the first ICU admission during the same hospital stay for a given patient. Second, we included only the first admission for each patient within a 30-day window to avoid including multiple readmissions that might occur in rapid succession for a single patient. Finally, we excluded patients enrolled in palliative care at the time of ICU admission.
Our primary outcome was mortality rates (ICU, in-hospital, and within 30 days of ICU admission), and our secondary outcomes were ICU and hospital LOS. First, we used bivariate methods (t test and χ2 statistic) to compare patient demographics, comorbid illness, admitting diagnosis, and severity during the pre-TM and post-TM periods in our intervention and control ICUs. We identified comorbid illnesses by applying algorithms developed by Elixhauser et al13 and Quan et al.14 We identified the primary condition associated with each ICU admission using algorithms from the Agency for Healthcare Research and Quality’s Clinical Classifications Software.15 Our principle analyses focused on all patients admitted to the intervention and control ICUs. We calculated predicted mortality rates for patients admitted to our intervention and control ICUs during the pre-TM and post-TM periods using models described in the eAppendix (Supplement). We compared predicted mortality rates across time (pre-TM and post-TM), as well as across intervention and control ICUs as a summary measure of patient complexity.
Second, we compared unadjusted ICU, in-hospital, and 30-day mortality rates during the pre-TM and post-TM periods in the intervention and control ICUs, as well as ICU and in-hospital LOS.
Third, we used mixed models to compare risk-adjusted mortality rates and LOS for patients hospitalized in the intervention and control ICUs (SAS software GLIMMIX procedure; SAS Institute). In particular we compared (1) pre-TM and post-TM mortality rates for the ICUs receiving the TM program; (2) pre-TM and post-TM mortality rates for the control ICUs (to evaluate secular trends in mortality rates in the control group); and (3) the magnitude of the change in mortality rates (pre-TM vs post-TM) in the intervention vs control ICUs based on interaction terms from our models. The dependent variable for our models was either mortality rate or LOS. Independent variables included patient demographics, comorbid illnesses, and primary diagnosis at ICU admission. We supplemented administrative data with selected laboratory test values to mirror risk-adjustment models developed for ICU patients (eg, Acute Physiology and Chronic Health Evaluation [APACHE] III scores). The C statistics for our mortality rate models (0.823-0.843) were generally similar to those for other VA ICU risk adjustment models.11,16
Fourth, we conducted an array of sensitivity analyses to examine the robustness of our findings. We examined the impact of the ICU TM in each intervention and control ICU individually, recognizing that our statistical power for these comparisons was markedly reduced. We stratified patients into 3 groups based on the predicted risk of death and examined the impact of TM on mortality rates in low-, intermediate-, and high-risk patients. We examined the impact of the ICU TM program in larger ICUs only (levels 1 and 2) and smaller ICUs only (levels 3 and 4); we also examined the impact of the TM program focusing exclusively on patients who required mechanical ventilation and those with sepsis. Finally, we repeated our analyses using an interrupted time series design as an alternative method for examining the potential impact of the TM program. Further details on our predicted mortality rate calculation and interrupted time series design are provided in the eAppendix (Supplement). All analyses were conducted using SAS software, version 9.3.
Our final analytical cohort consisted of 6939 ICU admissions (6654 patients), with 3355 admissions to intervention ICUs and 3584 admissions to control ICUs (Figure 1). Patient demographics, comorbid illness, and severity for the intervention and control ICUs for the pre-TM and post-TM periods are displayed in Table 1. For the intervention ICUs, patient demographics, admission diagnoses, comorbid illnesses, and predicted mortality rates were similar in the pre-TM and post-TM ICU TM periods. Likewise, there was little change in patient characteristics (Table 1) over the pre-TM and post-TM periods in the control ICUs. Viewing Table 1 from a different perspective, patients admitted to the intervention and control ICUs during the pre-TM and post-TM periods were generally similar. That said, predicted mortality was modestly lower in the intervention ICUs than in the control ICUs during the pre-TM period (3.0% vs 3.6%; P = .02) and post-TM period (2.8% vs 3.5%; P < .001), suggesting that patients treated in the control ICUs were modestly sicker.
Unadjusted mortality rates (ICU, in-hospital, and 30-day) and LOS (ICU and in-hospital) did not change significantly with implementation of the ICU TM program (Table 2). For example, ICU mortality in the intervention ICUs was 2.9% during the pre-TM period and 2.8% during the post-TM period (P = .89); findings were similar for in-hospital and 30-day mortality during the pre-TM and post-TM periods among the intervention ICUs. In analyses focusing on LOS in the intervention ICUs, we found no evidence of a decline in ICU LOS (2.6 vs 2.8 days; P = .15) or hospital LOS (6.9 vs 7.3 days; P = .18). Likewise, in analyses comparing pre-post differences in mortality rates in the intervention vs the control ICUs (Table 2), we found no evidence that mortality rates or LOS changed by a larger amount for patients treated in the intervention ICUs compared with those in the control ICUs.
In adjusted results, we found little evidence that the implementation of the ICU TM program resulted in a significant reduction in mortality rates or LOS when all ICUs were analyzed in aggregate (Table 3). Our findings of no statistically or clinically significant impact on mortality rates were consistent across ICU, in-hospital, and 30-day mortality rates. Likewise, results were consistent in our pre-post analyses focusing on the intervention ICUs in isolation (Table 3) and our analyses that compared the magnitude of change between the intervention ICUs and the concurrent control ICUs (Table 3). Results focusing on LOS yielded similar results.
We conducted several supplementary analyses to examine the robustness of our results. First, we examined the impact of the ICU TM program in each intervention and control ICU individually (Figure 2, Figure 3, and eTable 3 in the Supplement). These analyses demonstrated that intervention ICU 3 as well as control ICUs 7 and 8 had large reductions in ICU mortality rates (Figure 2 and eTable 3 in the Supplement). Interestingly, 30-day mortality results differed somewhat, with a number of control ICUs, but no intervention ICUs, having large reductions (Figure 3 and eTable 3 in the Supplement). Second, after stratifying patients based on their predicted risk of death (low, intermediate, or high), we failed to find evidence that the ICU TM program was more (or less) effective in higher- or lower-risk patient subgroups (eTable 4 in the Supplement). Third, we analyzed larger ICUs (levels 1 and 2 combined) compared with smaller ICUs (levels 3 and 4 combined). Again, we found no evidence that the ICU TM program was effective in either the larger or smaller ICUs in either unadjusted or adjusted analyses (eTables 5, 6, 7, and 8 in the Supplement). Fourth, we replicated our analyses using a segmented regression approach that incorporated hospital random intercepts and slopes to determine whether ICU TM implementation modified either the level or trend in risk-adjusted mortality rates. These analyses also showed no significant impact of TM implementation.
In a rigorous analysis of a large geographically dispersed network of hospitals within the VA health care system, we found no definitive evidence that the installation of an ICU TM program resulted in reductions in mortality rates or LOS. Our results provide important insight into the challenges of attaining the outcomes that are desired by policy makers, physicians, and administrators when implementing complex technologies. Beneath our aggregate results, analyses focusing on outcomes in individual ICUs suggest a far more complex picture. Our work also highlights the complex process of starting an ICU TM program in a large diverse health care system.
Some of our findings warrant further discussion. First, the overall lack of impact of an ICU TM program on mortality rates and LOS should be considered carefully. There are several potential explanations for this finding. One is that ICU TM simply does not reduce mortality rates. This possibility is supported by a number of studies, including those by Thomas et al8 and the systematic literature review and meta-analysis by Young et al.7 If readers were to draw such conclusions, ICU TM would join a number of other promising TM interventions that to date have shown limited effectiveness in clinical practice.17- 19
We would argue against such a strict interpretation of the data. Several studies, including a rigorous evaluation by Lilly et al,6,9 have demonstrated that ICU TM can reduce in-hospital and ICU mortality rates by 15% to 20%; these reductions are far larger than many medical and pharmacologic interventions that have been widely adopted by ICUs across the country. Discounting the potential impact of ICU TM prematurely could curtail the spread of a potentially enormously beneficial technology still in its infancy.
Rather, we suggest that the differences between the studies showing that ICU TM has no impact on mortality rates and those showing significant reductions in these rates can be attributed to a few key factors. The earliest studies of ICU TM explicitly noted that these programs needed to be much more than simple implementations of technology if they were to yield benefit.1,20 Lilly and colleagues9 were quite clear that a major component of their implementation was the effort devoted to systems reengineering in tandem with TM implementation. This reengineering included standardizing protocols across ICUs and instilling a strong culture of collaboration between bedside and TM care teams. Our experience with ICU TM implementation in the VA has shown us that the technology is necessary but far from sufficient.8,10,21
In considering our results, it is also important to note the between-site differences that we observed in both the adoption of the TM system and its impact. In particular, ICU TM hospital 3 (Figure 2) was observed to have rapidly accepted the TM system and integrated it into daily care.10 We suspect that this buy-in and integration can explain the significant reduction in ICU mortality rates in this particular ICU (P < .001) even when the program had no significant aggregate impact on mortality rates. Although we lack objective measures for the use of this technology at the individual-site level (eg, number of interactions and number of therapeutic recommendations made by the TM staff), our experience has been that not all sites adopted this technology equally. The degree of communication and level of acceptance of ICU TM varied according to the site and improved over time as the sites gained confidence and familiarity with the TM staff. A qualitative study of the ICU TM implementation process published by Moeckli et al10 revealed the challenges to successfully implementing the technology across a group of geographically dispersed ICUs. We believe that the variability in adopting this technology across the different ICUs may have prevented TM from reaching its full potential and contributed to the lack of benefit seen. These findings reinforce our strong belief in the critical importance of implementation and buy-in at the local level.
Second, we observed an extremely low mortality rate within our network compared with the private sector and the control ICUs selected from within the VA. We suspect that the low ICU mortality rate (2.9% in our study vs 10%-29% in other studies9,22,23) in our study is related to 3 factors: (1) the tendency of many VA hospitals to use ICU beds for monitoring and step-down functions24; (2) our exclusion of hospice patients from our cohort, in contrast to most other studies of ICU outcomes; and (3) the generally high attention to quality within the VA ICU community and our network in particular. The low baseline mortality rate in our intervention ICUs reduces our statistical power and makes it difficult to detect a statistically significant reduction in mortality rate (eTable 9 in the Supplement). It is equally reasonable to argue that if the ICU TM program cannot reduce ICU mortality or 30-day mortality by some modest amount (1%-2%) at a cost of $75 000 per bed per year, this program may not be cost-effective.25
Third, our results should lead us to consider whether mortality rates and LOS are the only appropriate outcomes for assessing the impact of ICU TM. Most of the highest-profile studies of ICU TM have focused on mortality rates because survival is arguably the most important end point to patients. However, a working group supported by the Agency for Healthcare Research and Quality explicitly recognized that ICU TM is a complex intervention that can affect an array of processes (eg, ventilator protocols and intravenous infusions), intermediate outcomes (eg, ventilator-associated pneumonia and ICU delirium), and final outcomes (eg, mortality rates) as well as system-level issues, such as interhospital transfer rates and use of palliative care.4 Much study is needed to evaluate the impact of ICU TM on these important outcomes as well.
Our study has numerous strengths that expand on prior studies. Our analysis was conducted in a geographically dispersed group of hospitals that included both tertiary and quaternary VA medical centers as well as small rural hospitals. It is, to our knowledge, the most rigorous analysis of ICU TM available in the current literature, using both a pre-post design and a matched control group. Furthermore, whereas most prior studies of mortality rates focused exclusively on ICU and/or in-hospital rates, our reliance on VA data files allowed us to focus on outcomes not affected by the hospital discharge bias (30-day mortality rates). We would argue that although ICU and in-hospital mortality rates are of interest, a longer-term perspective may be of greater interest to patients and physicians.
Our study has several limitations. First, it was conducted in a single integrated health care system in the upper Midwest, so extrapolation to other health care systems and settings should be done with care. Our results might not apply to non-VA hospitals or hospitals with higher mortality rates. Second, although our analysis focused on outcomes in patients admitted to the ICU, we did not examine how ICU TM might have affected ICU triage decisions or the transfer of patients between smaller and larger VA hospitals. Third, we did not examine the impact of the technology on code status or palliative care. Fourth, although the launch of the TM system was preceded by intensive education and site visits to facilitate transition, our analysis did not include a wash-in period. We will have follow-up data available, which will provide additional insight. Finally, we lacked detailed information about ICU TM use by the individual ICUs that implemented the monitoring system.
We found that implementation of an ICU TM program into a network of 7 hospitals did not reduce mortality rates or LOS. We suggest that hospitals considering implementation of ICU TM programs have modest expectations with respect to short-term impact on these outcomes. We also encourage such hospitals to allocate significant resources to the systems reengineering that has accompanied the most effective implementations that have had the largest impact.
Accepted for Publication: March 16, 2014.
Corresponding Author: Boulos S. Nassar, MD, MPH, Division of Pulmonary, Critical Care, and Occupational Medicine, Department of Internal Medicine, University of Iowa Hospitals and Clinics, 200 Hawkins Dr, Iowa City, IA 52242 (firstname.lastname@example.org).
Published Online: May 12, 2014. doi:10.1001/jamainternmed.2014.1503.
Author Contributions: Drs Nassar and Cram had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Nassar, Vaughan-Sarrazin, Reisinger, Cram,
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Nassar, Reisinger, Cram.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Nassar, Vaughan-Sarrazin, Jiang, Cram.
Obtained funding: Cram.
Administrative, technical, or material support: Nassar, Reisinger, Bonello, Cram.
Study supervision: Nassar, Vaughan-Sarrazin, Cram.
Conflict of Interest Disclosures: Dr Bonello is the medical director of the VISN 23 Tele-ICU program. No other disclosures are reported.
Funding/Support: This work is supported by a K24 award from the National Institute of Arthritis and Musculoskeletal and Skin Disease (grant AR062133 to Dr Cram) and by the US Department of Veterans Affairs (grant IRR 09-336 to all authors).
Role of the Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Disclaimer: The views expressed in this article are those of the authors and do not necessarily represent the views of the US Department of Veterans Affairs.