Development and Assessment of Objective Surveillance Definitions for Nonventilator Hospital-Acquired Pneumonia | Electronic Health Records | JAMA Network Open | JAMA Network
[Skip to Navigation]
Sign In
Figure.  Comparative Incidence and Hospital Mortality Rates in Tertiary and Community Hospitals for Selected Candidate Definitions for Nonventilator Hospital-Acquired Pneumonia
Comparative Incidence and Hospital Mortality Rates in Tertiary and Community Hospitals for Selected Candidate Definitions for Nonventilator Hospital-Acquired Pneumonia

Hospitals A and B represent the tertiary hospitals, and hospitals C and D represent the community hospitals. WBC indicates white blood cell.

Table 1.  Candidate Definitions for Hospital-Acquired Pneumonia in Nonventilated Patientsa
Candidate Definitions for Hospital-Acquired Pneumonia in Nonventilated Patientsa
Table 2.  Patient Characteristics
Patient Characteristics
Table 3.  Incidence and Potentially Attributable Length of Stay and Mortality for Hospital-Acquired Pneumonia in Nonventilated Patients
Incidence and Potentially Attributable Length of Stay and Mortality for Hospital-Acquired Pneumonia in Nonventilated Patients
Table 4.  Incidence of Selected Candidate Event Definitions by Clinical Service
Incidence of Selected Candidate Event Definitions by Clinical Service
1.
Magill  SS, Edwards  JR, Bamberg  W,  et al; Emerging Infections Program Healthcare-Associated Infections and Antimicrobial Use Prevalence Survey Team.  Multistate point-prevalence survey of health care–associated infections.  N Engl J Med. 2014;370(13):1198-1208. doi:10.1056/NEJMoa1306801PubMedGoogle ScholarCrossref
2.
Magill  SS, O’Leary  E, Janelle  SJ,  et al; Emerging Infections Program Hospital Prevalence Survey Team.  Changes in prevalence of health care–associated infections in U.S. hospitals.  N Engl J Med. 2018;379(18):1732-1744. doi:10.1056/NEJMoa1801550PubMedGoogle ScholarCrossref
3.
Baker  D, Quinn  B.  Hospital Acquired Pneumonia Prevention Initiative-2: incidence of nonventilator hospital-acquired pneumonia in the United States.  Am J Infect Control. 2018;46(1):2-7. doi:10.1016/j.ajic.2017.08.036PubMedGoogle ScholarCrossref
4.
Sopena  N, Sabrià  M; Neunos 2000 Study Group.  Multicenter study of hospital-acquired pneumonia in non-ICU patients.  Chest. 2005;127(1):213-219. doi:10.1378/chest.127.1.213PubMedGoogle ScholarCrossref
5.
See  I, Chang  J, Gualandi  N,  et al.  Clinical correlates of surveillance events detected by National Healthcare Safety Network pneumonia and lower respiratory tract definitions—Pennsylvania, 2011-2012.  Infect Control Hosp Epidemiol. 2016;37(7):818-824. doi:10.1017/ice.2016.74PubMedGoogle ScholarCrossref
6.
Micek  ST, Chew  B, Hampton  N, Kollef  MH.  A case-control study assessing the impact of nonventilated hospital-acquired pneumonia on patient outcomes.  Chest. 2016;150(5):1008-1014. doi:10.1016/j.chest.2016.04.009PubMedGoogle ScholarCrossref
7.
Russell  CD, Koch  O, Laurenson  IF, O’Shea  DT, Sutherland  R, Mackintosh  CL.  Diagnosis and features of hospital-acquired pneumonia: a retrospective cohort study.  J Hosp Infect. 2016;92(3):273-279. doi:10.1016/j.jhin.2015.11.013PubMedGoogle ScholarCrossref
8.
Corrado  RE, Lee  D, Lucero  DE, Varma  JK, Vora  NM.  Burden of adult community-acquired, health-care-associated, hospital-acquired, and ventilator-associated pneumonia: New York City, 2010 to 2014.  Chest. 2017;152(5):930-942. doi:10.1016/j.chest.2017.04.162PubMedGoogle ScholarCrossref
9.
Ibn Saied  W, Mourvillier  B, Cohen  Y,  et al; OUTCOMEREA Study Group.  A comparison of the mortality risk associated with ventilator-acquired bacterial pneumonia and nonventilator ICU-acquired bacterial pneumonia.  Crit Care Med. 2019;47(3):345-352. doi:10.1097/CCM.0000000000003553PubMedGoogle ScholarCrossref
10.
DiBiase  LM, Weber  DJ, Sickbert-Bennett  EE, Anderson  DJ, Rutala  WA.  The growing importance of non-device-associated healthcare-associated infections: a relative proportion and incidence study at an academic medical center, 2008-2012.  Infect Control Hosp Epidemiol. 2014;35(2):200-202. doi:10.1086/674847PubMedGoogle ScholarCrossref
11.
Davis  J, Finley  E.  The breadth of hospital-acquired pneumonia: nonventilated versus ventilated patients in Pennsylvania.  Pa Patient Saf Advis. 2012;9(3):99-105. http://patientsafety.pa.gov/ADVISORIES/Pages/201209_99.aspx. Accessed September 27, 2019.Google Scholar
12.
Hansen  S, Schwab  F, Zingg  W, Gastmeier  P; The Prohibit Study Group.  Process and outcome indicators for infection control and prevention in European acute care hospitals in 2011 to 2012—results of the PROHIBIT study.  Euro Surveill. 2018;23(21). doi:10.2807/1560-7917.ES.2018.23.21.1700513PubMedGoogle Scholar
13.
Horan  TC, Andrus  M, Dudeck  MA.  CDC/NHSN surveillance definition of health care–associated infection and criteria for specific types of infections in the acute care setting.  Am J Infect Control. 2008;36(5):309-332. doi:10.1016/j.ajic.2008.03.002PubMedGoogle ScholarCrossref
14.
Klompas  M.  Interobserver variability in ventilator-associated pneumonia surveillance.  Am J Infect Control. 2010;38(3):237-239. doi:10.1016/j.ajic.2009.10.003PubMedGoogle ScholarCrossref
15.
Stevens  JP, Kachniarz  B, Wright  SB,  et al.  When policy gets it right: variability in U.S. hospitals’ diagnosis of ventilator-associated pneumonia.  Crit Care Med. 2014;42(3):497-503. doi:10.1097/CCM.0b013e3182a66903PubMedGoogle ScholarCrossref
16.
Kerlin  MP, Trick  WE, Anderson  DJ,  et al.  Interrater reliability of surveillance for ventilator-associated events and pneumonia.  Infect Control Hosp Epidemiol. 2017;38(2):172-178. doi:10.1017/ice.2016.262PubMedGoogle ScholarCrossref
17.
Roulson  J, Benbow  EW, Hasleton  PS.  Discrepancies between clinical and autopsy diagnosis and the value of post mortem histology; a meta-analysis and review.  Histopathology. 2005;47(6):551-559. doi:10.1111/j.1365-2559.2005.02243.xPubMedGoogle ScholarCrossref
18.
Tejerina  E, Esteban  A, Fernández-Segoviano  P,  et al.  Accuracy of clinical definitions of ventilator-associated pneumonia: comparison with autopsy findings.  J Crit Care. 2010;25(1):62-68. doi:10.1016/j.jcrc.2009.05.008PubMedGoogle ScholarCrossref
19.
Roberts  IS, Benamore  RE, Benbow  EW,  et al.  Post-mortem imaging as an alternative to autopsy in the diagnosis of adult deaths: a validation study.  Lancet. 2012;379(9811):136-142. doi:10.1016/S0140-6736(11)61483-9PubMedGoogle ScholarCrossref
20.
van Mourik  MS, van Duijn  PJ, Moons  KG, Bonten  MJ, Lee  GM.  Accuracy of administrative data for surveillance of healthcare-associated infections: a systematic review.  BMJ Open. 2015;5(8):e008424. doi:10.1136/bmjopen-2015-008424PubMedGoogle ScholarCrossref
21.
Wolfensberger  A, Meier  AH, Kuster  SP, Mehra  T, Meier  MT, Sax  H.  Should International Classification of Diseases codes be used to survey hospital-acquired pneumonia?  J Hosp Infect. 2018;99(1):81-84. doi:10.1016/j.jhin.2018.01.017PubMedGoogle ScholarCrossref
22.
Young  M, Marrie  TJ.  Interobserver variability in the interpretation of chest roentgenograms of patients with possible pneumonia.  Arch Intern Med. 1994;154(23):2729-2732. doi:10.1001/archinte.1994.00420230122014PubMedGoogle ScholarCrossref
23.
Albaum  MN, Hill  LC, Murphy  M,  et al; PORT Investigators.  Interobserver reliability of the chest radiograph in community-acquired pneumonia.  Chest. 1996;110(2):343-350. doi:10.1378/chest.110.2.343PubMedGoogle ScholarCrossref
24.
Hopstaken  RM, Witbraad  T, van Engelshoven  JM, Dinant  GJ.  Inter-observer variation in the interpretation of chest radiographs for pneumonia in community-acquired lower respiratory tract infections.  Clin Radiol. 2004;59(8):743-752. doi:10.1016/j.crad.2004.01.011PubMedGoogle ScholarCrossref
25.
Dublin  S, Baldwin  E, Walker  RL,  et al.  Natural language processing to identify pneumonia from radiology reports.  Pharmacoepidemiol Drug Saf. 2013;22(8):834-841. doi:10.1002/pds.3418PubMedGoogle ScholarCrossref
26.
American Thoracic Society; Infectious Diseases Society of America.  Guidelines for the management of adults with hospital-acquired, ventilator-associated, and healthcare-associated pneumonia.  Am J Respir Crit Care Med. 2005;171(4):388-416. doi:10.1164/rccm.200405-644STPubMedGoogle ScholarCrossref
27.
Montravers  P, Veber  B, Auboyer  C,  et al.  Diagnostic and therapeutic management of nosocomial pneumonia in surgical patients: results of the Eole study.  Crit Care Med. 2002;30(2):368-375. doi:10.1097/00003246-200202000-00017PubMedGoogle ScholarCrossref
28.
Koulenti  D, Lisboa  T, Brun-Buisson  C,  et al; EU-VAP/CAP Study Group.  Spectrum of practice in the diagnosis of nosocomial pneumonia in patients requiring mechanical ventilation in European intensive care units.  Crit Care Med. 2009;37(8):2360-2368. doi:10.1097/CCM.0b013e3181a037acPubMedGoogle ScholarCrossref
29.
Esperatti  M, Ferrer  M, Giunta  V,  et al.  Validation of predictors of adverse outcomes in hospital-acquired pneumonia in the ICU.  Crit Care Med. 2013;41(9):2151-2161. doi:10.1097/CCM.0b013e31828a674aPubMedGoogle ScholarCrossref
30.
Magill  SS, Klompas  M, Balk  R,  et al.  Developing a new, national approach to surveillance for ventilator-associated events.  Crit Care Med. 2013;41(11):2467-2475. doi:10.1097/CCM.0b013e3182a262dbPubMedGoogle ScholarCrossref
31.
von Elm  E, Altman  DG, Egger  M, Pocock  SJ, Gøtzsche  PC, Vandenbroucke  JP; STROBE Initiative.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.  Ann Intern Med. 2007;147(8):573-577. doi:10.7326/0003-4819-147-8-200710160-00010PubMedGoogle ScholarCrossref
32.
Moore  BJ, White  S, Washington  R, Coenen  N, Elixhauser  A.  Identifying increased risk of readmission and in-hospital mortality using hospital administrative data: the AHRQ Elixhauser Comorbidity Index.  Med Care. 2017;55(7):698-705. doi:10.1097/MLR.0000000000000735PubMedGoogle ScholarCrossref
33.
Stuart  EA.  Matching methods for causal inference: a review and a look forward.  Stat Sci. 2010;25(1):1-21. doi:10.1214/09-STS313PubMedGoogle ScholarCrossref
34.
Naidus  EL, Lasalvia  MT, Marcantonio  ER, Herzig  SJ.  The diagnostic yield of noninvasive microbiologic sputum sampling in a cohort of patients with clinically diagnosed hospital-acquired pneumonia.  J Hosp Med. 2018;13(1):34-37. doi:10.12788/jhm.2868PubMedGoogle ScholarCrossref
35.
van der Eerden  MM, Vlaspolder  F, de Graaff  CS, Groot  T, Jansen  HM, Boersma  WG.  Value of intensive diagnostic microbiological investigation in low- and high-risk patients with community-acquired pneumonia.  Eur J Clin Microbiol Infect Dis. 2005;24(4):241-249. doi:10.1007/s10096-005-1316-8PubMedGoogle ScholarCrossref
36.
Giuliano  KK, Baker  D, Quinn  B.  The epidemiology of nonventilator hospital-acquired pneumonia in the United States.  Am J Infect Control. 2018;46(3):322-327. doi:10.1016/j.ajic.2017.09.005PubMedGoogle ScholarCrossref
37.
Nilsson  L, Risberg  MB, Montgomery  A, Sjödahl  R, Schildmeijer  K, Rutberg  H.  Preventable adverse events in surgical care in Sweden: a nationwide review of patient notes.  Medicine (Baltimore). 2016;95(11):e3047. doi:10.1097/MD.0000000000003047. Accessed September 27, 2019. PubMedGoogle ScholarCrossref
38.
Braykov  NP, Morgan  DJ, Schweizer  ML,  et al.  Assessment of empirical antibiotic therapy optimisation in six hospitals: an observational cohort study.  Lancet Infect Dis. 2014;14(12):1220-1227. doi:10.1016/S1473-3099(14)70952-1PubMedGoogle ScholarCrossref
39.
Fridkin  S, Baggs  J, Fagan  R,  et al; Centers for Disease Control and Prevention (CDC).  Vital signs: improving antibiotic use among hospitalized patients.  MMWR Morb Mortal Wkly Rep. 2014;63(9):194-200. https://www.cdc.gov/mmwr/preview/mmwrhtml/mm6309a4.htmPubMedGoogle Scholar
40.
Schurink  CAM, Nieuwenhoven  CAV, Jacobs  JA,  et al.  Clinical pulmonary infection score for ventilator-associated pneumonia: accuracy and inter-observer variability.  Intensive Care Med. 2004;30(2):217-224. doi:10.1007/s00134-003-2018-2PubMedGoogle ScholarCrossref
41.
Klompas  M.  Does this patient have ventilator-associated pneumonia?  JAMA. 2007;297(14):1583-1593. doi:10.1001/jama.297.14.1583PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    Infectious Diseases
    October 18, 2019

    Development and Assessment of Objective Surveillance Definitions for Nonventilator Hospital-Acquired Pneumonia

    Author Affiliations
    • 1Department of Pharmacy Administration and Clinical Pharmacy, School of Pharmacy, Xi’an Jiaotong University, Xi’an, Shaanxi, China
    • 2Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, Massachusetts
    • 3Department of Medicine, Brigham and Women’s Hospital, Boston, Massachusetts
    • 4Department of Medicine, Brigham and Women’s Faulkner Hospital, Boston, Massachusetts
    • 5Department of Medicine and Infection Control Unit, Massachusetts General Hospital, Boston
    JAMA Netw Open. 2019;2(10):e1913674. doi:10.1001/jamanetworkopen.2019.13674
    Key Points español 中文 (chinese)

    Question  Is it possible to conduct operational surveillance using the clinical data routinely recorded in electronic health records to identify nonventilated adults with hospital-acquired pneumonia?

    Findings  In this cohort study of 310 651 patients with 489 519 admissions, an electronic surveillance definition based on worsening oxygenation, at least 3 days of new antibiotics, fever or abnormal white blood cell count, and performance of chest imaging was successfully applied to all patients. This definition identified 0.6 event per 100 admissions and was associated with up to a 6-fold higher risk of hospital death compared with matched control patients.

    Meaning  This study suggests that electronic surveillance for nonventilator hospital-acquired pneumonia is feasible; this approach could inform the development and evaluation of pneumonia prevention programs in hospitals.

    Abstract

    Importance  Hospital-acquired pneumonia is the most common health care–associated infection in the United States. Most cases occur in nonventilated patients, but many hospitals track hospital-acquired pneumonia only in ventilated patients because of the complexity and subjectivity of conducting surveillance for large numbers of nonventilated patients.

    Objective  To propose and assess potentially objective, efficient, and reproducible surveillance definitions for nonventilator hospital-acquired pneumonia (NV-HAP) using routine clinical data stored in electronic health record systems.

    Design, Setting, and Participants  This cohort study was conducted in 2 tertiary referral and 2 community hospitals in Massachusetts between May 31, 2015, and July 1, 2018. All nonventilated patients aged 18 years or older who were admitted to these hospitals were included (N = 310 651).

    Exposures  Ten candidate definitions for NV-HAP based on clinically meaningful combinations of 6 potential surveillance criteria were proposed: worsening oxygenation, temperature higher than 38 °C (fever), abnormal white blood cell count of less than 4000/μL or more than 12 000/μL, performance of chest imaging, submission of respiratory specimen for culture, and 3 or more days of new antibiotics.

    Main Outcomes and Measures  Incidence rates, lengths of stay, hospital mortality rates, and odds ratios (ORs) for time to discharge and mortality compared with those of matched controls were calculated for each candidate definition. The ORs were adjusted for demographics, clinical service, comorbidities, and severity of illness.

    Results  The study analyzed 310 651 patients with 489 519 admissions, including 205 054 patients with 311 484 admissions of 3 or more days. Among the patients with 311 484 admissions, the mean (SD) patient age was 58.3 (19.3) years and 176 936 (56.8%) were of women. Incidence rates for candidate definitions per 100 admissions ranged from 3.4 events for worsening oxygenation alone to 0.9 event for worsening oxygenation and at least 3 days of new antibiotics to 0.6 event for worsening oxygenation, at least 3 days of new antibiotics, fever, abnormal white blood cell count, and performance of chest imaging. Crude mortality rates ranged from 16.1% (n = 2643) for patients with worsening oxygen alone to 27.7% (n = 868) for patients with worsening oxygenation, at least 3 days of antibiotics, fever or abnormal white blood cell count, and chest imaging. Patients who met NV-HAP candidate definitions remained in the hospital for twice as long as their matched controls (adjusted ORs ranged from 1.8 [95% CI, 1.7-1.8] to 2.1 [95% CI, 2.0-2.1]) and were 4 to 6 times as likely to die in the hospital (adjusted ORs ranged from 3.8 [95% CI, 3.5-4.0] to 6.5 [95% CI, 5.2-8.2]). Agreement between candidate definitions and clinical diagnoses was fair (κ = 0.33).

    Conclusions and Relevance  These findings suggest that objective surveillance for NV-HAP using electronically computable definitions that incorporate common clinical criteria is feasible and generates incidence, mortality, and adjusted ORs for hospital mortality similar to estimates from manual surveillance. These definitions have the potential to facilitate widespread, automated surveillance for NV-HAP and thus inform the development and evaluation of prevention programs.

    Introduction

    Hospital-acquired pneumonia is the most common health care–associated infection in the United States, affecting approximately 1% of hospitalized patients, two-thirds of whom are not receiving mechanical ventilation when pneumonia begins.1-3 Crude mortality rates for patients with nonventilator hospital-acquired pneumonia (NV-HAP) range between 15% and 30%, which are similar to or higher than the rates associated with ventilator-associated pneumonia.3-9 Most hospitals, however, do not have formal surveillance or prevention programs for NV-HAP.3,10-12

    One of the major barriers to creating surveillance and prevention programs for NV-HAP is the complexity and subjectivity of the surveillance definitions published by the US Centers for Disease Control and Prevention (CDC).13 They require radiographic evidence of a “new or progressive and persistent infiltrate”; systemic signs, such as fever, abnormal white blood cell (WBC) count, or altered mental status; and pulmonary signs, such as rales, new cough, “new onset of purulent sputum, or change in character of sputum,” or “worsening gas exchange.”13(p330) These criteria are subjective, nonspecific, and difficult to abstract because they are inconsistently documented and typically require reading free-text notes rather than analyzing structured data.14-16 The challenge is magnified by the size of the population at risk. In contrast to ventilator-associated pneumonia surveillance, almost every patient in the hospital must be tracked for NV-HAP rather than just the relatively small number of patients receiving mechanical ventilation. Furthermore, the correlation between clinical criteria and histological pneumonia is poor, and interobserver variability is high.17-19 Some researchers have proposed using administrative data for surveillance rather than clinical data, but diagnosis codes compound the limited accuracy of clinical criteria because they add coders’ interpretations to clinicians’ variable and imperfect impressions. The sensitivity of coding for NV-HAP is only between 40% and 60%, and the positive predictive value is between 30% and 50%.3,20,21

    The limitations of current pneumonia definitions make routine surveillance for NV-HAP difficult for hospitals to conduct. This difficulty, in turn, limits hospitals’ capabilities to develop and evaluate pneumonia prevention programs. New surveillance definitions for NV-HAP are needed; ideally, these should be definitions that can be electronically applied using the data routinely found in electronic health record (EHR) systems. This would allow for the possibility of efficient, objective, and reproducible NV-HAP surveillance for large populations. Electronically computable definitions are unlikely to overcome the limited accuracy of clinical signs for pneumonia, but definitions that mirror clinical diagnostic criteria and that reliably and reproducibly detect nosocomial events associated with poor outcomes may provide a credible alternative to manual surveillance using the CDC traditional criteria.13

    Given the potential value of an electronically computable surveillance definition for NV-HAP, we undertook a systematic evaluation of potential definitions using clinical data routinely found in most modern EHR systems. We identified potential clinical components to include in surveillance definitions; proposed candidate definitions using these components; and characterized the incidence, overlap with clinically diagnosed NV-HAP, and associations with time to discharge and hospital mortality for each candidate definition across 4 hospitals.

    Methods
    Study Design and Data Sources

    We developed 10 candidate definitions for NV-HAP and then retrospectively applied them to all nonventilated patients aged 18 years or older who were admitted to 2 academic and 2 community hospitals in eastern Massachusetts (Brigham and Women’s Hospital, Massachusetts General Hospital, Newton-Wellesley Hospital, and Faulkner Hospital) between May 31, 2015, and July 1, 2018. Data for analysis were derived from the Epic EHR (Epic Systems Corp) using the Partners HealthCare Research Patient Data Repository, Enterprise Data Warehouse, and Microbiology databases. The study was reviewed and approved by the Partners HealthCare Institutional Review Board, which issued a waiver of informed consent because the study was retrospective, was observational, and presented results in aggregate. We followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guidelines.31

    Candidate Definitions

    We identified clinical indicators routinely measured or obtained in pneumonia diagnosis that are (1) available for most patients in EHR systems, (2) objective, and (3) recorded as structured data. This yielded 6 possible clinical indicators: worsening oxygenation, new antibiotic administration, fever, abnormal WBC count, performance of chest imaging, and submission of respiratory specimens for culture. We did not include radiographic interpretations because they are subjective and nonspecific and require special expertise in natural language processing to parse that is not available in most hospitals.19,22-25 We also did not include arterial blood gases because they are measured in only a small subset of patients.

    Next, we proposed 10 potential candidate definitions for NV-HAP surveillance that were based on clinically meaningful combinations of these 6 clinical indicators (Table 1). We required worsening oxygenation as an anchor sign for all candidate definitions because impaired oxygenation is a cardinal manifestation of pneumonia.26-29 All additional criteria included in each candidate definition (eg, fever, abnormal WBC count, and antibiotic starts) were required to be present on the first or second day of worsening oxygenation. Candidate definitions were modeled after the objective components of the CDC traditional surveillance criteria for hospital-acquired pneumonia but with adaptations similar to those implemented by the CDC to create ventilator-associated event definitions.13,30

    We defined worsening oxygenation as 2 or more days of stable or improving oxygenation followed by 2 or more days of any of the following: (1) a decrease in daily maximum oxygen saturation from 95% or higher in a patient on ambient air to less than 95% on ambient air, (2) the initiation of supplemental oxygen, (3) an increase in daily median oxygen flow rate of at least 3 L/min for a nasal cannula and at least 4 L/min for a simple face mask, or (4) an escalation in oxygen delivery device (nasal cannula → shovel mask → simple face mask → oxygen conservation device → high-flow oxygen → non-rebreather → noninvasive positive pressure ventilation → intubation and mechanical ventilation). The criteria for escalation of supplemental oxygen for different baseline states of oxygenation are further described in eTable 1 in the Supplement.

    We defined fever as a daily maximum temperature higher than 38 °C and abnormal WBC count as more than 12 000/μL or less than 4000/μL (to convert to ×109 per liter, multiply by 0.001). We counted both plain chest radiographs and computed tomography of the chest as eligible radiographic criteria. We defined new antibiotic start as antibiotics administered on hospital day 3 or later that were not given on the previous 2 calendar days. We required at least 3 days of new antibiotics to focus surveillance on the subset of potential pneumonias that clinicians deemed serious enough to merit sustained treatment. Antibiotics were limited to agents used to treat pneumonia (eTable 2 in the Supplement). New antibiotics started and continued for only 1 or 2 days were acceptable if the patient died before receiving a 3-day course. Antibiotic changes on the second or third day of treatment were permitted as long as there were no gaps of a calendar day or more in antibiotic administrations.

    Outcomes

    We calculated incidence rates, time to discharge, and hospital mortality rates for each NV-HAP candidate definition. We included more than 1 potential NV-HAP event in a single admission for incidence estimates as long as the events were more than 14 days apart. However, we included only 1 randomly chosen event per admission to estimate the potential excess time to discharge and hospital mortality rate for each candidate definition. Incidence rates were expressed as the number of NV-HAP events per 100 admissions and per 1000 hospital days.

    Statistical Analysis

    Data were analyzed from August 30, 2018, through August 23, 2019. We estimated potential excess time to discharge and hospital mortality rate by separately matching patients with each candidate definition with up to 4 randomly selected control patients. We required control patients to be drawn from the same hospital and clinical service as the case patients and to be hospitalized for at least as long as the time until the case patient’s second day of impaired oxygenation, which provided parity with our requirement for case patients to have at least 2 days of impaired oxygenation. Case patients were eligible to be controls for other cases as long as they were not yet cases as of the day of matching. We estimated the potential excess time to discharge using negative binomial regression as the rate ratio of the mean count of days from NV-HAP onset to hospital discharge across case patients divided by the mean count of days from match to hospital discharge across control patients. Potential excess mortality rate was estimated using logistic regression as the odds ratio (OR) of hospital death in cases compared with the controls.

    We calculated crude and adjusted ORs with 95% CIs for all models. We adjusted for hospital, clinical service, age, sex, race/ethnicity, comorbidities (congestive heart failure, chronic lung disease, pulmonary vascular disease, diabetes, liver disease, kidney disease, neurological disease, stroke, lymphoma, and solid malignant neoplasm), Elixhauser Comorbidity Index,32 time to NV-HAP (match date), WBC count, hemoglobin level, hematocrit level, platelet count, serum creatinine, blood urea nitrogen level, sodium level, aspartate aminotransferase level, alanine aminotransferase level, total bilirubin level, albumin level, international normalized ratio, and history of antibiotic exposure between admission and day before meeting a given candidate definition. Missing laboratory values were imputed using the last known value or normal if never checked. We used robust SEs with generalized estimating equations to account for correlations within matched sets.33 Comparisons for each definition were estimated separately. All analyses were performed using SAS, version 9.4 (SAS Institute Inc).

    High hospital mortality rates among patients who met NV-HAP candidate definitions may bias toward the null the excess time-to-discharge estimates among patients with NV-HAP. Therefore, we conducted a sensitivity analysis of time to discharge on survivors only. We also conducted sensitivity analyses including only patients with complete laboratory data and patients in whom respiratory cultures were obtained.

    Clinically Diagnosed NV-HAP

    We evaluated the sensitivity, specificity, and positive predictive value of the combination of worsening oxygenation, new antibiotics, fever or abnormal WBC count, and chest imaging to identify clinically suspected NV-HAP through medical record reviews. We focused on this definition because it mirrors the existing CDC criteria for pneumonia and infection-related ventilator-associated complications.13,30 We randomly selected the medical records of 120 patients hospitalized for 3 or more days who met electronic surveillance criteria for impaired oxygenation. We nested the medical record reviews among patients with impaired oxygenation partly because of sampling efficiency, given the rarity of NV-HAP among undifferentiated patients, and partly because impaired oxygenation is a cardinal sign for pneumonia; the absence of impaired oxygenation makes pneumonia unlikely.26-29

    One of us (H.R.B), who was blinded to the selection criteria, reviewed medical records for clinical documentation of suspected pneumonia (as defined by treating clinicians). Another one of us (M.K.) independently reviewed 10% of the medical records to confirm consistency and accuracy.

    Results

    In total, 310 651 patients with 489 519 admissions were identified during the study period, including 205 054 patients with 311 484 admissions of 3 or more days. Among the patients with 311 484 admissions, the mean (SD) patient age was 58.3 (19.3) years and 176 936 (56.8%) were of women. The characteristics of all 311 484 admissions and patients who met a subset of candidate definitions are presented in Table 2. Data on the completeness of laboratory data and the success of matching are provided in eTable 3 in the Supplement.

    The incidence of NV-HAP varied by candidate definition, ranging from 3.4 events per 100 admissions for patients with worsening oxygenation alone to 0.9 event per 100 admissions for patients with worsening oxygenation and at least 3 days of new antibiotics to 0.6 event per 100 admissions for patients with worsening oxygenation, at least 3 days of antibiotics, fever or abnormal WBC count, and performance of chest imaging (Table 3). Radiographic studies were obtained in 3795 patients (83.8%) with worsening oxygenation and antibiotics, but pulmonary cultures were obtained in only 1186 patients (26.2%) with worsening oxygenation and antibiotics. Given the low frequency of pulmonary cultures among patients who otherwise appeared to have possible NV-HAP, we focused on candidate definitions without this requirement.

    The median (interquartile range [IQR]) hospital LOS among all patients admitted for 3 or more days was 5 (4-8) days. In contrast, the median (IQR) LOS for patients with NV-HAP ranged from 14 (9-24) days for the worsening oxygenation definition alone to 17 (11-29) days for worsening oxygenation and at least 3 days of new antibiotics to 19 (12-31) days for worsening oxygenation, at least 3 days of new antibiotics, fever or abnormal WBC count, and chest imaging. The adjusted ORs for time from NV-HAP to discharge in cases compared with controls ranged from 2.1 (95% CI, 2.0-2.1) for worsening oxygenation alone to 1.8 (95% CI, 1.7-1.8) for worsening oxygenation and at least 3 days of new antibiotics to 1.8 (95% CI, 1.7-1.9) for worsening oxygenation, at least 3 days of new antibiotics, fever or abnormal WBC count, and chest imaging. Sensitivity analyses limited to survivors yielded similar results (eTable 4 in the Supplement).

    The crude hospital mortality rate among all patients hospitalized at least 3 days was 2.4% (n = 7475). In contrast, the crude mortality rates for patients who met candidate NV-HAP criteria ranged from 16.1% (n = 2643) for patients with worsening oxygenation alone to 23.9% (n = 1081) for patients with worsening oxygenation and at least 3 days of new antibiotics to 27.7% (n = 868) for patients with worsening oxygenation, at least 3 days of new antibiotics, fever or abnormal WBC count, and chest imaging. The adjusted ORs for hospital death in cases vs controls ranged from 3.8 (95% CI, 3.5-4.0) for patients with worsening oxygenation alone to 5.1 (95% CI, 4.6-5.8) for worsening oxygenation and at least 3 days of new antibiotics to 6.3 (95% CI, 5.5-7.2) for worsening oxygenation, at least 3 days of new antibiotics, fever or abnormal WBC count, and chest imaging, to 6.5 (95% CI, 5.2-8.2) for patients who met all the potential surveillance criteria. Sensitivity analyses restricted to patients with complete data and patients in whom pulmonary cultures were obtained generated similar results (eTable 5 in the Supplement).

    Of the 120 medical records reviewed, 42 patients (35.0%) met the potential NV-HAP definition of worsening oxygenation, at least 3 days of new antibiotics, fever or abnormal WBC count, and chest imaging. Agreement between this candidate definition and clinicians’ working diagnoses was fair (κ = 0.33). Of the 42 patients flagged by this definition, treating clinicians documented concern for pneumonia in 25 patients (59.5%), with a positive predictive value of 60% (95% CI, 47%-71%). The potential reasons for deterioration in oxygenation for the remaining 17 patients (40.5%) included pulmonary edema, atelectasis, aspiration, pulmonary embolism, periprocedural airway management, and sepsis of nonpulmonary origin. Clinicians documented concern for pneumonia in 20 additional patients who did not meet the candidate electronic definition for NV-HAP, thus yielding a net sensitivity of 56% (95% CI, 40%-70%) and specificity of 77% (95% CI, 68%-86%).

    Incidence rates varied widely between clinical services (Table 4). By the candidate definition of worsening oxygenation, at least 3 days of new antibiotics, fever or abnormal WBC count, and chest imaging, potential NV-HAP events per 100 admissions were most common in cardiac surgery (3.3 events), cardiology (1.2 events), oncology (1.7 events), and intensive care (2.9 events) units. Events were least common in the obstetrics (0 events) and gynecology (0.2 event) services. Incidence rates were higher in the 2 tertiary referral hospitals compared with the 2 community hospitals, but mortality rates were similar in all hospitals (Figure).

    Discussion

    Nonventilator hospital-acquired pneumonia is among the most common health care–associated infections and is associated with high morbidity and mortality. Yet the complexity and subjectivity of traditional surveillance definitions have prevented most hospitals from implementing active surveillance and prevention programs for NV-HAP. This study demonstrated the feasibility of conducting surveillance using electronically computable surveillance definitions that mirror the clinical diagnosis and care patterns for NV-HAP, generate incidence rates similar to previously published manually generated estimates, and are associated with prolonged times to discharge and increased hospital mortality rates. These electronically computable events thus provide a potential means to support operational surveillance for NV-HAP and to inform the development of prevention and monitoring programs.

    We proposed 10 candidate definitions for NV-HAP, ranging from worsening oxygenation alone to worsening oxygenation, fever, abnormal WBC count, performance of chest imaging, submission of pulmonary specimens for culture, and at least 3 days of new antibiotics. Incidence rates for each definition varied inversely with the number of surveillance criteria, with a sharp decrease in incidence for candidate definitions that required respiratory cultures. This decrease is consistent with previous observations that clinicians only obtain pulmonary specimens from a minority of patients with symptoms suggestive of pneumonia.34,35

    A potentially credible surveillance definition, then, might be one that includes worsening oxygenation, at least 3 days of new antibiotics, fever or abnormal WBC count, and obtaining chest imaging. The incidence rate for this definition was 0.6 event per 100 admissions. This rate matches other investigators’ estimates of the prevalence of NV-HAP, including 2 multistate point-prevalence surveys conducted by the CDC that also estimated 0.6 event per 100 patients.1-3,8,36,37

    Mortality rates for the candidate definitions mirrored the 15% to 30% mortality rates reported by other studies.3-8 We observed crude mortality rates ranging from 16.1% for worsening oxygenation alone to 27.7% for the combination of worsening oxygenation, at least 3 days of new antibiotics, fever or abnormal WBC count, and chest imaging. Similarly, patients who developed NV-HAP approximately doubled their time to discharge compared with patients who did not develop NV-HAP.

    We found that the odds of hospital death for NV-HAP were high even after adjusting for a wide array of potential confounders, including hospital, clinical service, age, comorbidities, laboratory results, premorbid LOS, and previous antibiotic exposure. We estimated that patients with NV-HAP were 5 to 6 times more likely to die in the hospital compared with patients without NV-HAP. This finding mirrors the results from other studies. Micek and colleagues,6 for example, estimated an adjusted OR of death of 8.4 (95% CI, 5.6-12.5) for patients with NV-HAP.

    Agreement between the proposed candidate definitions and clinical notations for pneumonia was only fair (κ = 0.33). Diagnosing pneumonia, however, is notoriously subjective and inaccurate.17-21,38-40 Many other studies have documented similarly low rates of interobserver agreement among infection preventionists (κ range = 0.12-0.40) and among clinicians (κ range = 0.16-0.34) who diagnose pneumonia.14-16,40 Furthermore, the positive predictive value of a clinical diagnosis of NV-HAP is only 30% to 50%.3,20,21 The significance of the mismatch between electronic surveillance criteria and clinical documentation is therefore unclear. Electronically computable definitions that use objective criteria may overcome some of the variability and inconsistencies inherent in traditional surveillance, but we have no reason to believe that they are any more (or less) accurate than traditional surveillance definitions.

    Future studies are needed to prospectively validate the proposed surveillance definitions, further assess their correlation with clinical events labeled by physicians, identify risk factors, and measure their preventability by instituting best practices to prevent hospital-acquired pneumonia.

    Strengths and Limitations

    The inclusion of antibiotic prescription criteria in the candidate definitions is both a strength and a weakness. It is a strength because it gives credence to clinicians’ impressions and limits detection to events that clinicians deemed serious and certain enough to warrant a sustained course of antibiotics. It is a weakness because clinicians vary in their predilection to diagnose and treat pneumonia, antibiotics may be administered for something other than pneumonia, overuse of antibiotics to treat possible pneumonia is common, and previous studies suggest that clinical impressions correlate poorly with biopsy or autopsy.17,18,38,39,41 The worsening oxygenation requirement, in contrast, likely adds some specificity to the definition, given that pulmonary impairment is a sine qua non for clinically meaningful pneumonia.26-29

    This study has several limitations. First, it was conducted in 4 hospitals from a single region in the United States. Although the findings were similar across all study hospitals and matched independent estimates by other investigators, the results may differ in other clinical settings with distinct patient populations and clinical practice patterns. Second, the estimates of potential excess time to discharge and mortality rate may be biased by residual confounding, despite adjusting for a wide array of demographic and clinical factors. Third, we assessed only the association between surveillance definitions and clinically diagnosed pneumonia in patients with worsening oxygenation. This approach may have led to an overestimation of sensitivity. The accuracy and meaningfulness of clinically diagnosed pneumonia are questionable, however, in patients on ambient air with oxygen saturations consistently above 95%.26-29 Similarly, we reviewed only a small number of medical records, and thus the estimates of clinical correlates have wide confidence intervals. Fourth, the proposed candidate surveillance definitions are amenable for implementation only in hospital systems with comprehensive EHRs that include vital signs, demographics, antibiotic administrations, radiographic procedures, clinical cultures, and laboratory values as well as analytic support to extract and analyze these data. Disseminating this surveillance strategy will likely require sharing common data specifications and analytic code with hospitals and/or incorporating these definitions into commercial infection surveillance software systems.

    Conclusions

    This study proposed objective surveillance definitions for NV-HAP that can be electronically computed using routine EHR data. These candidate definitions yielded incidence and mortality rates comparable to existing estimates based on manual surveillance. All definitions were associated with longer time to discharge and higher mortality rates. These definitions may therefore provide hospitals with an efficient and objective means to conduct routine surveillance for NV-HAP and thus to develop and evaluate prevention programs.

    Back to top
    Article Information

    Accepted for Publication: September 3, 2019.

    Published: October 18, 2019. doi:10.1001/jamanetworkopen.2019.13674

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2019 Ji W et al. JAMA Network Open.

    Corresponding Author: Michael Klompas, MD, MPH, Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, 401 Park Dr, Ste 401 East, Boston, MA 02215 (mklompas@bwh.harvard.edu).

    Author Contributions: Dr Klompas had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Ji, Ramirez Batlle, Zhang, Rhee, Klompas.

    Acquisition, analysis, or interpretation of data: All authors.

    Drafting of the manuscript: Ji, McKenna, Ramirez Batlle, Zhang, Klompas.

    Critical revision of the manuscript for important intellectual content: Ochoa, Young, Zhang, Rhee, Clark, Shenoy, Hooper, Klompas.

    Statistical analysis: Ji, Ochoa, Ramirez Batlle, Young, Zhang, Klompas.

    Obtained funding: Klompas.

    Administrative, technical, or material support: Ochoa, Clark, Shenoy.

    Supervision: Klompas.

    Conflict of Interest Disclosures: Ms Ochoa reported receiving grants from the Centers for Disease Control and Prevention (CDC) during the conduct of the study. Dr Young reported receiving grants from the CDC during the conduct of the study. Dr Rhee reported receiving grants from Agency for Healthcare Research and Quality and royalties from UpToDate outside the submitted work. Dr Klompas reported receiving grants from the CDC during the conduct of the study and royalties from UpToDate outside the submitted work. No other disclosures were reported.

    Funding/Support: This study was funded by the CDC Prevention Epicenters Program.

    Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Additional Contributions: Kenton Plourde, BSBA, Brigham and Women’s Hospital, as well as Karen Lynch, RN, and Karina Bradford, MSEE, Massachusetts General Hospital, assisted with assembling the raw data to support this study. These individuals received no compensation for their contributions.

    References
    1.
    Magill  SS, Edwards  JR, Bamberg  W,  et al; Emerging Infections Program Healthcare-Associated Infections and Antimicrobial Use Prevalence Survey Team.  Multistate point-prevalence survey of health care–associated infections.  N Engl J Med. 2014;370(13):1198-1208. doi:10.1056/NEJMoa1306801PubMedGoogle ScholarCrossref
    2.
    Magill  SS, O’Leary  E, Janelle  SJ,  et al; Emerging Infections Program Hospital Prevalence Survey Team.  Changes in prevalence of health care–associated infections in U.S. hospitals.  N Engl J Med. 2018;379(18):1732-1744. doi:10.1056/NEJMoa1801550PubMedGoogle ScholarCrossref
    3.
    Baker  D, Quinn  B.  Hospital Acquired Pneumonia Prevention Initiative-2: incidence of nonventilator hospital-acquired pneumonia in the United States.  Am J Infect Control. 2018;46(1):2-7. doi:10.1016/j.ajic.2017.08.036PubMedGoogle ScholarCrossref
    4.
    Sopena  N, Sabrià  M; Neunos 2000 Study Group.  Multicenter study of hospital-acquired pneumonia in non-ICU patients.  Chest. 2005;127(1):213-219. doi:10.1378/chest.127.1.213PubMedGoogle ScholarCrossref
    5.
    See  I, Chang  J, Gualandi  N,  et al.  Clinical correlates of surveillance events detected by National Healthcare Safety Network pneumonia and lower respiratory tract definitions—Pennsylvania, 2011-2012.  Infect Control Hosp Epidemiol. 2016;37(7):818-824. doi:10.1017/ice.2016.74PubMedGoogle ScholarCrossref
    6.
    Micek  ST, Chew  B, Hampton  N, Kollef  MH.  A case-control study assessing the impact of nonventilated hospital-acquired pneumonia on patient outcomes.  Chest. 2016;150(5):1008-1014. doi:10.1016/j.chest.2016.04.009PubMedGoogle ScholarCrossref
    7.
    Russell  CD, Koch  O, Laurenson  IF, O’Shea  DT, Sutherland  R, Mackintosh  CL.  Diagnosis and features of hospital-acquired pneumonia: a retrospective cohort study.  J Hosp Infect. 2016;92(3):273-279. doi:10.1016/j.jhin.2015.11.013PubMedGoogle ScholarCrossref
    8.
    Corrado  RE, Lee  D, Lucero  DE, Varma  JK, Vora  NM.  Burden of adult community-acquired, health-care-associated, hospital-acquired, and ventilator-associated pneumonia: New York City, 2010 to 2014.  Chest. 2017;152(5):930-942. doi:10.1016/j.chest.2017.04.162PubMedGoogle ScholarCrossref
    9.
    Ibn Saied  W, Mourvillier  B, Cohen  Y,  et al; OUTCOMEREA Study Group.  A comparison of the mortality risk associated with ventilator-acquired bacterial pneumonia and nonventilator ICU-acquired bacterial pneumonia.  Crit Care Med. 2019;47(3):345-352. doi:10.1097/CCM.0000000000003553PubMedGoogle ScholarCrossref
    10.
    DiBiase  LM, Weber  DJ, Sickbert-Bennett  EE, Anderson  DJ, Rutala  WA.  The growing importance of non-device-associated healthcare-associated infections: a relative proportion and incidence study at an academic medical center, 2008-2012.  Infect Control Hosp Epidemiol. 2014;35(2):200-202. doi:10.1086/674847PubMedGoogle ScholarCrossref
    11.
    Davis  J, Finley  E.  The breadth of hospital-acquired pneumonia: nonventilated versus ventilated patients in Pennsylvania.  Pa Patient Saf Advis. 2012;9(3):99-105. http://patientsafety.pa.gov/ADVISORIES/Pages/201209_99.aspx. Accessed September 27, 2019.Google Scholar
    12.
    Hansen  S, Schwab  F, Zingg  W, Gastmeier  P; The Prohibit Study Group.  Process and outcome indicators for infection control and prevention in European acute care hospitals in 2011 to 2012—results of the PROHIBIT study.  Euro Surveill. 2018;23(21). doi:10.2807/1560-7917.ES.2018.23.21.1700513PubMedGoogle Scholar
    13.
    Horan  TC, Andrus  M, Dudeck  MA.  CDC/NHSN surveillance definition of health care–associated infection and criteria for specific types of infections in the acute care setting.  Am J Infect Control. 2008;36(5):309-332. doi:10.1016/j.ajic.2008.03.002PubMedGoogle ScholarCrossref
    14.
    Klompas  M.  Interobserver variability in ventilator-associated pneumonia surveillance.  Am J Infect Control. 2010;38(3):237-239. doi:10.1016/j.ajic.2009.10.003PubMedGoogle ScholarCrossref
    15.
    Stevens  JP, Kachniarz  B, Wright  SB,  et al.  When policy gets it right: variability in U.S. hospitals’ diagnosis of ventilator-associated pneumonia.  Crit Care Med. 2014;42(3):497-503. doi:10.1097/CCM.0b013e3182a66903PubMedGoogle ScholarCrossref
    16.
    Kerlin  MP, Trick  WE, Anderson  DJ,  et al.  Interrater reliability of surveillance for ventilator-associated events and pneumonia.  Infect Control Hosp Epidemiol. 2017;38(2):172-178. doi:10.1017/ice.2016.262PubMedGoogle ScholarCrossref
    17.
    Roulson  J, Benbow  EW, Hasleton  PS.  Discrepancies between clinical and autopsy diagnosis and the value of post mortem histology; a meta-analysis and review.  Histopathology. 2005;47(6):551-559. doi:10.1111/j.1365-2559.2005.02243.xPubMedGoogle ScholarCrossref
    18.
    Tejerina  E, Esteban  A, Fernández-Segoviano  P,  et al.  Accuracy of clinical definitions of ventilator-associated pneumonia: comparison with autopsy findings.  J Crit Care. 2010;25(1):62-68. doi:10.1016/j.jcrc.2009.05.008PubMedGoogle ScholarCrossref
    19.
    Roberts  IS, Benamore  RE, Benbow  EW,  et al.  Post-mortem imaging as an alternative to autopsy in the diagnosis of adult deaths: a validation study.  Lancet. 2012;379(9811):136-142. doi:10.1016/S0140-6736(11)61483-9PubMedGoogle ScholarCrossref
    20.
    van Mourik  MS, van Duijn  PJ, Moons  KG, Bonten  MJ, Lee  GM.  Accuracy of administrative data for surveillance of healthcare-associated infections: a systematic review.  BMJ Open. 2015;5(8):e008424. doi:10.1136/bmjopen-2015-008424PubMedGoogle ScholarCrossref
    21.
    Wolfensberger  A, Meier  AH, Kuster  SP, Mehra  T, Meier  MT, Sax  H.  Should International Classification of Diseases codes be used to survey hospital-acquired pneumonia?  J Hosp Infect. 2018;99(1):81-84. doi:10.1016/j.jhin.2018.01.017PubMedGoogle ScholarCrossref
    22.
    Young  M, Marrie  TJ.  Interobserver variability in the interpretation of chest roentgenograms of patients with possible pneumonia.  Arch Intern Med. 1994;154(23):2729-2732. doi:10.1001/archinte.1994.00420230122014PubMedGoogle ScholarCrossref
    23.
    Albaum  MN, Hill  LC, Murphy  M,  et al; PORT Investigators.  Interobserver reliability of the chest radiograph in community-acquired pneumonia.  Chest. 1996;110(2):343-350. doi:10.1378/chest.110.2.343PubMedGoogle ScholarCrossref
    24.
    Hopstaken  RM, Witbraad  T, van Engelshoven  JM, Dinant  GJ.  Inter-observer variation in the interpretation of chest radiographs for pneumonia in community-acquired lower respiratory tract infections.  Clin Radiol. 2004;59(8):743-752. doi:10.1016/j.crad.2004.01.011PubMedGoogle ScholarCrossref
    25.
    Dublin  S, Baldwin  E, Walker  RL,  et al.  Natural language processing to identify pneumonia from radiology reports.  Pharmacoepidemiol Drug Saf. 2013;22(8):834-841. doi:10.1002/pds.3418PubMedGoogle ScholarCrossref
    26.
    American Thoracic Society; Infectious Diseases Society of America.  Guidelines for the management of adults with hospital-acquired, ventilator-associated, and healthcare-associated pneumonia.  Am J Respir Crit Care Med. 2005;171(4):388-416. doi:10.1164/rccm.200405-644STPubMedGoogle ScholarCrossref
    27.
    Montravers  P, Veber  B, Auboyer  C,  et al.  Diagnostic and therapeutic management of nosocomial pneumonia in surgical patients: results of the Eole study.  Crit Care Med. 2002;30(2):368-375. doi:10.1097/00003246-200202000-00017PubMedGoogle ScholarCrossref
    28.
    Koulenti  D, Lisboa  T, Brun-Buisson  C,  et al; EU-VAP/CAP Study Group.  Spectrum of practice in the diagnosis of nosocomial pneumonia in patients requiring mechanical ventilation in European intensive care units.  Crit Care Med. 2009;37(8):2360-2368. doi:10.1097/CCM.0b013e3181a037acPubMedGoogle ScholarCrossref
    29.
    Esperatti  M, Ferrer  M, Giunta  V,  et al.  Validation of predictors of adverse outcomes in hospital-acquired pneumonia in the ICU.  Crit Care Med. 2013;41(9):2151-2161. doi:10.1097/CCM.0b013e31828a674aPubMedGoogle ScholarCrossref
    30.
    Magill  SS, Klompas  M, Balk  R,  et al.  Developing a new, national approach to surveillance for ventilator-associated events.  Crit Care Med. 2013;41(11):2467-2475. doi:10.1097/CCM.0b013e3182a262dbPubMedGoogle ScholarCrossref
    31.
    von Elm  E, Altman  DG, Egger  M, Pocock  SJ, Gøtzsche  PC, Vandenbroucke  JP; STROBE Initiative.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.  Ann Intern Med. 2007;147(8):573-577. doi:10.7326/0003-4819-147-8-200710160-00010PubMedGoogle ScholarCrossref
    32.
    Moore  BJ, White  S, Washington  R, Coenen  N, Elixhauser  A.  Identifying increased risk of readmission and in-hospital mortality using hospital administrative data: the AHRQ Elixhauser Comorbidity Index.  Med Care. 2017;55(7):698-705. doi:10.1097/MLR.0000000000000735PubMedGoogle ScholarCrossref
    33.
    Stuart  EA.  Matching methods for causal inference: a review and a look forward.  Stat Sci. 2010;25(1):1-21. doi:10.1214/09-STS313PubMedGoogle ScholarCrossref
    34.
    Naidus  EL, Lasalvia  MT, Marcantonio  ER, Herzig  SJ.  The diagnostic yield of noninvasive microbiologic sputum sampling in a cohort of patients with clinically diagnosed hospital-acquired pneumonia.  J Hosp Med. 2018;13(1):34-37. doi:10.12788/jhm.2868PubMedGoogle ScholarCrossref
    35.
    van der Eerden  MM, Vlaspolder  F, de Graaff  CS, Groot  T, Jansen  HM, Boersma  WG.  Value of intensive diagnostic microbiological investigation in low- and high-risk patients with community-acquired pneumonia.  Eur J Clin Microbiol Infect Dis. 2005;24(4):241-249. doi:10.1007/s10096-005-1316-8PubMedGoogle ScholarCrossref
    36.
    Giuliano  KK, Baker  D, Quinn  B.  The epidemiology of nonventilator hospital-acquired pneumonia in the United States.  Am J Infect Control. 2018;46(3):322-327. doi:10.1016/j.ajic.2017.09.005PubMedGoogle ScholarCrossref
    37.
    Nilsson  L, Risberg  MB, Montgomery  A, Sjödahl  R, Schildmeijer  K, Rutberg  H.  Preventable adverse events in surgical care in Sweden: a nationwide review of patient notes.  Medicine (Baltimore). 2016;95(11):e3047. doi:10.1097/MD.0000000000003047. Accessed September 27, 2019. PubMedGoogle ScholarCrossref
    38.
    Braykov  NP, Morgan  DJ, Schweizer  ML,  et al.  Assessment of empirical antibiotic therapy optimisation in six hospitals: an observational cohort study.  Lancet Infect Dis. 2014;14(12):1220-1227. doi:10.1016/S1473-3099(14)70952-1PubMedGoogle ScholarCrossref
    39.
    Fridkin  S, Baggs  J, Fagan  R,  et al; Centers for Disease Control and Prevention (CDC).  Vital signs: improving antibiotic use among hospitalized patients.  MMWR Morb Mortal Wkly Rep. 2014;63(9):194-200. https://www.cdc.gov/mmwr/preview/mmwrhtml/mm6309a4.htmPubMedGoogle Scholar
    40.
    Schurink  CAM, Nieuwenhoven  CAV, Jacobs  JA,  et al.  Clinical pulmonary infection score for ventilator-associated pneumonia: accuracy and inter-observer variability.  Intensive Care Med. 2004;30(2):217-224. doi:10.1007/s00134-003-2018-2PubMedGoogle ScholarCrossref
    41.
    Klompas  M.  Does this patient have ventilator-associated pneumonia?  JAMA. 2007;297(14):1583-1593. doi:10.1001/jama.297.14.1583PubMedGoogle ScholarCrossref
    ×