Electronic health record (EHR)-based alerts are often used to notify practitioners of abnormal test results, but follow-up failures (missed results) continue to occur in outpatient settings.1 In the Department of Veterans Affairs (VA), abnormal test result alerts are generated automatically for prespecified abnormal laboratory values or manually by the interpreting radiologist when an unexpected finding is noted.2-4 Factors such as workflow, user behaviors, and organizational characteristics likely affect EHR-based test result follow-up.1,5 Thus, we examined the “sociotechnical” predictors of missed test results in the setting of EHR-based alerts.
From June 2010 through November 2010, we conducted a cross-sectional survey of VA primary care practitioners (PCPs); trainees and subspecialists were excluded.6 The survey content was informed by an 8-dimensional sociotechnical model7 representing multiple complex facets of EHR-based test result notification. Survey items assessed PCPs' perceptions of technological factors (eg, EHR notification software, its ease of use, content of alerts PCPs received, EHR user interface) and social factors (eg, workflow, people, and organizational policies and procedures) related to alert follow-up. We assessed potential information overload by asking if practitioners received more alerts than they could effectively manage or received too many alerts to focus on the most important ones.
After pilot testing, we administered the 105-item survey using a web-based survey administration service. To increase response rates, invitation e-mails and reminders were followed by telephone attempts to reach nonrespondents. In accordance with VA policies, we did not use monetary incentives for participation.
We defined primary outcomes related to missed test results based on respondents' answers to 2 items: outcome 1 (potential for missed results) based on “The alert notification system in Computerized Patient Record System (CPRS) as currently implemented makes it possible for practitioners to miss test results,” and outcome 2 (personal history of missed results) based on “In the past year, I missed abnormal lab or imaging test results that led to delayed patient care.”
We examined correlation coefficients to determine significant relationships between sociotechnical variables and each primary outcome. Variables significantly related to the outcomes in bivariate analyses were then included in multivariate regression analyses. For ease of interpretation, we subsequently recoded the outcome responses into dichotomous categories and conducted logistic regression analyses. Responses of “agree” or “strongly agree” were coded as affirmative. Independent variables in the logistic regression models were those that were significant for both outcomes in the initial linear models. Stepwise selection was used to identify predictors significantly related to the outcome (P ≤ .05).
Of 5001 PCPs invited, 2590 (51.8%) responded. The median number of alerts PCPs reported receiving each day was 63; 86.9% perceived the quantity of alerts they received to be excessive, and 69.6% reported receiving more alerts than they could effectively manage (marker of information overload).
Over half (55.6%) reported that the EHR notification system as currently implemented made it possible for practitioners to miss test results. Almost a third (29.8%) reported having personally missed results that led to care delays.
The Table indicates the significant predictors in logistic regression. Perceived ease of EHR use was related to a lower likelihood of both the perception of potentially missing results (odds ratio [OR], 0.52 [95% CI, 0.32-0.86]) and to reporting missed results that led to care delays (OR, 0.64 [95% CI, 0.43-0.96]). Greater concern over electronic hand-offs (ie, routing alerts to the EHR of a surrogate covering practitioner) was also related to potential for (OR, 2.00 [95% CI, 1.38-2.89]) and personal history of (OR, 1.86 [95% CI, 1.28-2.69]) missed test results. PCPs who reported receiving more alerts than was manageable (information overload) were more likely to report having missed results that led to delayed patient care (OR, 2.20 [95% CI, 1.37-3.52]). Notably, the number of alerts that respondents reported they received per day was unrelated to either outcome.
Our data suggest that PCPs using comprehensive EHRs are vulnerable to information overload, which might lead them to miss important information. While the alert quantity data was self-reported, it was strikingly similar to what we found using objective methods querying the EHR of a single VA facility (mean, 56.4 per day per PCP).8
Our study also suggests an association between usability and patient safety in the context of missed results. Because the EHR and its user interface is the same across all VA facilities, these perceptions were likely affected by other sociotechnical contextual factors that affect EHR-based test result follow-up. Current efforts to measure and improve usability should encompass a broad, “real-world” context that includes broader sociotechnical facets of the work environment.7,9 Moreover, an isolated reduction in alert numbers without attention to the broader PCP experience related to information overload might be insufficient to improve outcomes.10
Because this was a cross-sectional survey, we cannot determine causation. Nevertheless, our findings suggest that missed results in EHRs might be related to information overload from alert notifications, electronic hand-offs in care, and practitioner perceptions of poor EHR usability. Interventions to improve safety of test result follow-up in EHRs must address these factors.
Correspondence: Dr Singh, Michael E. DeBakey Veterans Affairs Medical Center (152), 2002 Holcombe Blvd, Houston, TX 77030 (hardeeps@bcm.edu).
Published Online: March 4, 2013. doi:10.1001/2013.jamainternmed.61
Author Contributions: Dr Singh had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Singh, Spitzmueller, Sawhney, and Sittig. Acquisition of data: Singh and Spitzmueller. Analysis and interpretation of data: Singh, Spitzmueller, Petersen, Sawhney, and Sittig. Drafting of the manuscript: Singh, Spitzmueller, and Sittig. Critical revision of the manuscript for important intellectual content: Singh, Spitzmueller, Petersen, Sawhney, and Sittig. Statistical analysis: Spitzmueller and Petersen. Obtained funding: Singh. Administrative, technical, and material support: Singh and Sittig. Study supervision: Singh and Sittig.
Conflict of Interest Disclosures: None reported.
Funding/Support: The research reported herein was supported by the Department of Veterans Affairs National Center for Patient Safety, a National Institutes of Health K23 career development award (K23CA125585) to Dr Singh and in part by the Houston VA Health Services Research and Development Center of Excellence (HFP90-020). Dr Sittig is supported in part by a SHARP contract from the Office of the National Coordinator for Health Information Technology (ONC No. 10510592).
Additional Contributions: We thank additional members of our project team, Donna Espadas, BS; Daniel Murphy, MD, MBA; Michael Smith, PhD; and Archana Laxmisan, MD, MA (all affiliated with the Houston VA Health Services Research and Development Center of Excellence), for their valuable contributions to the survey. None of them received additional compensation for this work.
Disclaimer: The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs or any other funding agencies.
1.Callen JL, Westbrook JI, Georgiou A, Li J. Failure to follow-up test results for ambulatory patients: a systematic review [published online December 11, 2011].
J Gen Intern Med22183961
PubMedGoogle Scholar 2.Singh H, Thomas EJ, Mani S,
et al. Timely follow-up of abnormal diagnostic imaging test results in an outpatient setting: are electronic medical records achieving their potential?
Arch Intern Med. 2009;169(17):1578-158619786677
PubMedGoogle ScholarCrossref 3.Singh H, Thomas EJ, Sittig DF,
et al. Notification of abnormal lab test results in an electronic medical record: do any safety concerns remain?
Am J Med. 2010;123(3):238-24420193832
PubMedGoogle ScholarCrossref 4.Singh H, Vij MS. Eight recommendations for policies for communicating abnormal test results.
Jt Comm J Qual Patient Saf. 2010;36(5):226-23220480756
PubMedGoogle Scholar 5.Hysong SJ, Sawhney MK, Wilson L,
et al. Understanding the management of electronic test result notifications in the outpatient setting.
BMC Med Inform Decis Mak. 2011;11:2221486478
PubMedGoogle ScholarCrossref 6.Singh H, Spitzmueller C, Petersen NJ,
et al. Primary care practitioners' views on test result management in EHR-enabled health systems: a national survey [published online December 24, 2012].
J Am Med Inform AssocGoogle Scholar 7.Sittig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems.
Qual Saf Health Care. 2010;19:(suppl 3)
i68-i7420959322
PubMedGoogle ScholarCrossref 8.Murphy DR, Reis B, Sittig DF, Singh H. Notifications received by primary care practitioners in electronic health records: a taxonomy and time analysis.
Am J Med. 2012;125(2):209- e1-e722269625
PubMedGoogle ScholarCrossref 9.Roth EM, Eggelston RG. Forging new evaluation paradigms: beyond statistical generalization. In: Patterson E, Miller J, eds. Macrocognition Metrics and Scenarios: Design and Evaluation for Real-World Teams. Burlington, VT: Ashgate Publishing; 2010:203-219
10.Singh H, Wilson L, Reis B, Sawhney MK, Espadas D, Sittig DF. Ten strategies to improve management of abnormal test result alerts in the electronic health record.
J Patient Saf. 2010;6(2):121-12320563228
PubMedGoogle ScholarCrossref