Antibiotic Timing and Errors in Diagnosing Pneumonia

Background  The percentage of patients with community-acquired pneumonia (CAP) whose time to first antibiotic dose (TFAD) is less than 4 hours of presentation to the emergency department (ED) has been made a core quality measure, and public reporting has been instituted. We asked whether these time pressures might also have negative effects on the accuracy of diagnosis of pneumonia.

Methods  We performed a retrospective review of adult admissions for CAP for 2 periods: group 1, when the core quality measure was a TFAD of less than 8 hours; and group 2, when the TFAD was lowered to less than 4 hours. We examined the accuracy of diagnosis of CAP by ED physicians.

Results  A total of 548 patients diagnosed as having CAP were studied (255 in group 1 and 293 in group 2). At admission, group 2 patients were 39.0% less likely to meet predefined diagnostic criteria for CAP than were group 1 patients (odds ratio, 0.61; 95% confidence interval, 0.42-0.86) (P = .004). At discharge, there was agreement between the ED physician's diagnosis and the predefined criteria for CAP in 62.0% of group 1 and 53.9% of group 2 patients (P = .06) and between the ED physician's admitting diagnosis and that of the discharging physician in 74.5% of group 1 and 66.9% of group 2 patients (P = .05). The mean (SD) TFAD was similar in group 1 (167.0 [118.6] minutes) and group 2 (157.8 [96.3] minutes).

Conclusion  Reduction in the required TFAD from 8 to 4 hours seems to reduce the accuracy by which ED physicians diagnose pneumonia, while failing to reduce the actual TFAD achieved for patients.

A “door-to-needle” time of less than 4 hours for the empirical administration of antibiotics for community-acquired pneumonia (CAP) is one of the core measures by which hospitals and physicians are evaluated for the quality of their care by the Centers for Medicare & Medicaid Services, the Hospital Quality Alliance, and the Joint Commission¹ (data available at http://www.jointcommission.org/PerformanceMeasurement/ [accessed February 11, 2007]). The percentage of patients with CAP with a time to first antibiotic dose (TFAD) within 4 hours of presentation is publicly reported for individual hospitals on the Internet at sites such as http://www.hospitalcompare.hhs.gov/ (accessed February 11, 2007). Hospitals with lower scores may suffer in reputation and reimbursement,² which may create a sense of urgency for making a diagnosis and instituting antibiotic therapy in emergency department (ED) patients with respiratory symptoms.^3-5

This decision to make TFAD for CAP a core quality measure was based on retrospective analyses^6-9 that showed improved outcomes in patients who received antibiotics within 8 or 4 hours of presentation. Time pressures to reduce TFAD, however, can lead to errors in diagnosis that could result in unnecessary antibiotic use for patients who do not have CAP, premature closure in decision making, and biasing of subsequent clinical problem solving.¹⁰ Potential gains in morbidity and mortality by prompt antibiotic therapy could be counterbalanced by inappropriate treatment of patients misdiagnosed as having CAP and by delays and added costs in making the correct diagnosis. Indeed, the excessive use of a hospital's new pneumonia care plan may have contributed to a severe outbreak of Clostridium difficile infection. Medical record review of this outbreak identified that half of the patients being treated for pneumonia may not have had pneumonia.¹¹ These unintended consequences of performance measurements must be considered when governmental agencies choose how to measure and reward institutions.

We describe a retrospective analysis of patients admitted to a medium-size community teaching hospital with a diagnosis of CAP in 2 periods of time: one period in which the TFAD quality standard was less than 8 hours and one period after the TFAD standard had been lowered to less than 4 hours. We asked whether the pressure to decrease the TFAD from 8 to 4 hours might adversely affect accuracy of diagnosis and could paradoxically increase overall morbidity of patients empirically treated for CAP.

A retrospective medical record analysis included patients older than 18 years who were diagnosed by the ED physician as having pneumonia and who were admitted to Franklin Square Hospital Center from November 1, 2003, through April 30, 2004 (group 1), and from November 1, 2004, through April 30, 2005 (group 2). These periods were chosen to correspond to the Maryland Healthcare Association's quality initiatives, in which the recommended TFAD for pneumonia was 8 and 4 hours from arrival at the hospital, respectively. In addition, an educational initiative regarding the new 4-hour TFAD expectation and responsibility for achieving it was given to the ED physicians in October 2004. Because we were aware that this period experiences hospital censuses exceeding capacity and that patients have an increased prevalence of respiratory symptoms that may influence the diagnostic accuracy rates, we controlled for this effect by choosing identical periods during the years studied for our comparison groups. Franklin Square Hospital Center is a 365-bed university-affiliated community teaching hospital in a socioeconomically diverse area of Baltimore, with approximately 100 000 ED visits each year, of which nearly 8000 result in admission to the acute care medical wards.

Patients with pneumonia who were admitted to the hospital through the ED were identified according to the ED physicians' admission diagnoses of pneumonia. However, an ED patient care associate is responsible for entering the ED physician's admission diagnosis into the hospital database (a potential source of error). Once entered, this admission diagnosis cannot be edited. Patients were identified via a query of this hospital admission diagnosis database using the search terms pneumonia, CAP, and NHAP (nursing home–acquired pneumonia). From an alphabetical list of all patients with an admission diagnosis of pneumonia, every other patient was chosen for the medical record review. Patients with more than 1 hospitalization for pneumonia during one of the specified 6-month periods had only their first hospitalization during that period reviewed. Data were abstracted from electronic and paper medical records by 2 attending physicians (J.A.W. and M.H.) and 2 medical students supervised by those same physicians. Length of stay and charges were obtained from the database of CareScience (Philadelphia, Pennsylvania). Interobserver variability was minimized by the use of an Internet database with electronic tools that identified potentially inaccurate entries and the senior author's review (J.A.W.) of each record for completeness and accuracy before locking the database.

The primary outcome measured was to test the null hypothesis that the accuracy of the ED physicians' admission diagnoses of pneumonia would not change before and after the core quality measure reduction of the goal: TFAD from less than 8 hours to less than 4 hours from arrival at the ED. The secondary outcome was to test the null hypothesis that efforts to administer antibiotics within 4 hours of admission would not have an adverse effect in patients who were misdiagnosed as having pneumonia by the ED physicians during the 2 periods. Examining whether earlier administration of antibiotics was beneficial in patients who were accurately diagnosed as having pneumonia was not an outcome measured in this study.

Precise criteria for the diagnosis of pneumonia are not available and had to be developed. We based our criteria on those found in Food and Drug Administration–directed clinical trials. Pneumonia was considered present when the medical record documented the presence of all 3 of the following criteria: (1) the presence of a new or increasing infiltrate by chest radiograph or computed tomographic scan; (2) plus a temperature greater than 38.0°C or less than 35.1°C or a total white blood cell count greater than 10/μL or less than 4.5/μL (to convert to cell count × 10⁹/L, multiply by 0.001), or immature polymorphonuclear leukocytes (bands) greater than 15%; and (3) plus 2 of cough, dyspnea, pleuritic chest pain, tachypnea with respirations of 30/min or greater, hypoxia with pulse oximetry of less than 90% or PaO₂ of less than 60 mm Hg, auscultatory findings of pneumonia, including rales, dullness of percussion, bronchial breath sounds, or egophony, or newly required mechanical ventilation by either intubation or noninvasive ventilation.

The primary outcome was tested in 3 ways: (1) Did the patient have a diagnosis of pneumonia at discharge? (2) Did the patient meet our predetermined diagnostic criteria for the diagnosis of pneumonia at admission? (3) At discharge, did the patient's entire medical record support the admission diagnosis of pneumonia according to our predefined criteria?

Secondary outcomes were assessed by examining antibiotic-associated adverse events, including allergic reactions, iatrogenic injuries, and associated morbidity and mortality; length of stay and charges; and delays in documenting the patient's correct diagnosis. Morbidity was broadly defined as any adverse outcome that resulted from a treatment either provided or withheld while the patient was incorrectly diagnosed as having CAP. This includes a reaction or adverse effect to a medication given for CAP or an outcome that is likely the result of the delay in treatment (ie, an antibiotic-associated adverse drug reaction or increasing ventilator support requirements of a patient with congestive heart failure who was given intravenous fluids rather than diuretics). A diagnostic delay was recorded when the diagnosis that persisted through hospital discharge was recorded in the medical record by a physician more than 24 hours after the admission history and physical were completed. Because all patients were admitted by a medical resident, a hospitalist, or a physician's assistant, the medical history and physical examination were routinely completed within 6 hours of admission. All patients were seen by an attending physician within 24 hours and typically in less than 8 hours of arrival to the ED. Delay in diagnosis was considered not to be present if the correct diagnosis was documented anywhere in the medical record in the first 24 hours of hospitalization or in any of the admission history and physical notes on the first or second days of hospitalization.

The sample size determination used a commercially available software program (SPSS for Windows; SPSS Inc, Chicago, Illinois [available at http://www.spss.com/spss/ and accessed February 11, 2005]) to calculate a required sample size of 393 subjects. This is assuming α = .05 (accepted type I error), power = 0.80, δ = 0.10 (detectable difference would be a 10% change in misdiagnosis rate), σ = 0.5 (standard deviation within a sample population), and m = 1 (ratio of study-control population). This also assumed the use of an independent t test. The calculation offered an 80% chance of detecting an increase in error of admitting diagnosis that might be attributable to efforts to treat patients in less than 4 hours. Based on a preliminary assessment that demonstrated that, on average, 3 to 4 adults are admitted daily with pneumonia during the winter at Franklin Square Hospital Center, an estimated 500 or more study subjects were expected within each of the 6-month periods from November through April.

Statistical analysis of baseline characteristics and outcomes included χ² analysis for categorical data. Continuous data were compared by t test, unless there was a large discrepancy from normal distribution, in which case a Mann-Whitney test was performed. The nonparametric Mann-Whitney test was required for the average time to antibiotics, average doses of antibiotics, average number of days patients received antibiotics, length of stay, and charges.

The total study population comprised 627 patients with an ED admission diagnosis of pneumonia, 286 who were admitted between November 1, 2003, and April 31, 2004 (group 1), and 341 who were admitted between November 1, 2004, and April 31, 2005 (group 2). Our medical record review, however, revealed that only 255 (89.2%) of the 2003-2004 patients and 293 (85.9%) of the 2004-2005 patients had actually been diagnosed as having CAP by the ED physician. This discrepancy represents a data entry error rate of 10.8% and 14.1% by our ED patient care associates, respectively, which was not statistically significantly different for the 2 periods. We also did a random sampling of 100 admissions of patients who arrived during the 2 periods and had a diagnosis assigned by the ED patient care associate other than CAP, and found no instances in which the ED physician had actually diagnosed CAP. The CAP admissions we studied constituted 255 of 7060 (3.6%) and 293 of 7743 (3.8%) of the total ED admissions during the 2 fiscal periods. The annualized estimate of total numbers of admissions for CAP (applying the 89.2% and 85.9% correction factors for incorrect assignment of diagnosis by ED patient care associates) was 1020 of 7060 (14.4%) and 1172 of 7743 (15.1%) for fiscal years 2004 and 2005, respectively.

The baseline characteristics of the 2 patient populations were similar at presentation to the ED (Table 1), and there was no difference in TFAD between group 1 and group 2.

The agreement of the ED admission diagnosis with predefined criteria for CAP (Table 2) was poor in group 1 (<8-hour TFAD expectation) and group 2 (<4-hour TFAD expectation) patients: when compared with predefined diagnostic criteria for CAP at admission, the ED physician's diagnosis was accurate 45.9% of the time from November 2003 to April 2004 and 33.8% of the time from November 2004 to April 2005. The odds ratio of the ED physician's diagnosis satisfying the predefined diagnostic criteria for CAP at admission was 0.61 (95% confidence interval, 0.42-0.86) when comparing group 2 with group 1 patients, indicating that admissions were 39.0% less likely to meet predefined diagnostic criteria for CAP when diagnosed during the less than 4-hour TFAD as opposed to the less than 8-hour TFAD.

When the agreement with predefined criteria for CAP was assessed using all information available at discharge, the ED physician's diagnosis met the predefined criteria for CAP 62.0% of the time in group 1 and 53.9% of the time in group 2 patients, a difference that approached statistical significance (Table 2). When comparing agreement between the ED physician's admitting diagnosis and that of the discharging physician, there was congruence in 74.5% of group 1 and 66.9% of group 2 patients. Notably, most of the non-CAP discharge diagnoses are illnesses that receive no benefit from CAP antibiotics (Table 3).

Patients who were admitted with a diagnosis of CAP but discharged with another diagnosis were more likely to experience a delay in diagnosis in group 2 compared with group 1: 20.1% vs 13.3% (P = .03). Analysis of the mortality and morbidity of the 232 patients who were judged not to have had CAP at discharge when the predefined criteria were applied (Table 4) revealed no significant differences between the 2 groups. While there are no benchmarks for comparison, it is notable that 28.0% of these patients experienced a delay in the establishment of the correct diagnosis.

We examined the accuracy of diagnosis of CAP by ED physicians, as well as patient care outcomes, during 2 periods just before and just after the Centers for Medicare & Medicaid Services/Joint Commission standard for the treatment of CAP was tightened from a TFAD of less than 8 hours to a TFAD of less than 4 hours for patients admitted from the ED with a diagnosis of CAP. In particular, we asked whether the behavioral result of pressures to improve a TFAD, that might already have been optimal, could have undermined any positive outcomes. We were concerned that the new TFAD was inadvertently creating a time pressure by adding a deadline to the physicians' decision-making process and thereby influencing the quality of their decisions. As Metersky et al¹³ noted, when hospital reputation and pay-for-performance are at stake, only 100% adherence can ensure an institution is not outperformed by others.

We demonstrated that only 70.4% of 548 patients with an ED admission diagnosis of CAP were discharged with a primary or secondary diagnosis of CAP, 39.4% had signs and symptoms that fulfilled the predefined criteria for CAP at admission, and 42.3% had clinical findings of CAP that met the predefined criteria at discharge. Therefore, depending on the rigor with which one assigns the diagnosis of CAP, nearly one-third to two-thirds of patients who were admitted and empirically treated for CAP seem not to have had that diagnosis at discharge. We also showed that accuracy of diagnosis of CAP by the ED physician, whether judged by predefined criteria or by the discharging physician's diagnosis, deteriorated in the period that had a less than 4-hour TFAD goal (group 2) (Table 2).

The mean (SD) TFAD was similar and substantially less than the core measure goals for group 1 (167.0 [118.6] minutes vs the goal of 480 minutes) and group 2 (157.8 [96.3] minutes vs the goal of 240 minutes); the percentage of patients who received antibiotics in less than 4 hours did show a minor increase from 81.3% to 85.6% that was not statistically significant. We believe that the 2 TFADs may well represent the best achievable in our ED and that the pressure to achieve a shorter TFAD in group 2 patients reduced the accuracy of diagnosis by ED physicians without significantly shortening the TFAD (Table 2).

While instituting antibiotic therapy for CAP earlier likely generates better patient outcomes, the consequences of pressuring physicians and hospitals to administer antibiotics in less than 4 hours have not been tested prospectively. Consider that a TFAD of less than 4 hours has only been shown to benefit patients older than 65 years who have not received treatment before arrival at the ED.⁵ While for hospital operations it is understandable to use this goal for all patients, from a quality viewpoint one must consider the unintended consequences of the universal application of this performance measurement. Our study adds to other recent studies^13,14 that challenge unquestioning acceptance of the validity and expectations set on hospitals of this well-meaning, but perhaps hastily implemented, criterion for quality of care. Furthermore, by identifying that 10 of 232 patients (4.3%) incorrectly diagnosed as having pneumonia will have an antibiotic-associated adverse drug reaction or other morbidity following a delay in arriving at the correct diagnosis, our study suggests that anecdotes of patients experiencing morbidity while incorrectly labeled as having CAP may be valid (Table 4).

The fundamental problem in any study of pneumonia is that there are no gold standards or universally accepted criteria for the diagnosis of CAP. Consider the association of a prolonged hospital length of stay with an increased TFAD.¹⁵ In their study, patients were considered to have CAP based on the diagnosis of the ED physician, and the influence of the decreased diagnostic accuracy we found associated with a decreased TFAD is unknown. For example, if patients with a shorter TFAD were more likely to have bronchitis rather than CAP, this may shorten this group's length of stay. A strength of our study is that we examined outcomes for patients meeting predefined criteria for the diagnosis of CAP based on requirements in Food and Drug Administration–approved studies for enrollment of patients into trials of new antibiotics for respiratory tract infections. Prior studies have typically required only a physician's working diagnosis of pneumonia at admission and a chest radiograph indicating a new infiltrate.¹⁶ We believe that these are inadequate diagnostic criteria for CAP for modern hospital medicine. It was our experience in enrolling patients into clinical trials that the predefined criteria we used were, in fact, flexible, reasonable, and accurate diagnostic standards. We question, moreover, whether most patients who do not fulfill these criteria can legitimately be considered to have a secure enough diagnosis of pneumonia to justify treatment with empirical antibiotics for that diagnosis.

Other strengths of our study are that the cases were drawn from an urban and a suburban population that was racially and socioeconomically balanced and that considerable effort was put into ensuring accuracy of data collection. Although we took steps to minimize interobserver reliability, the absence of statistics documenting this result is a study weakness. Another weakness is the retrospective design of this study; as Houck³ points out, however, it may no longer be ethically possible to perform a prospective study of timing of antibiotics for CAP.

Recently, the appropriateness of a performance target of 100% has been challenged by Metersky et al,¹³ and the scientific basis for asserting that a TFAD is a modifiable predictor of CAP-related morbidity and mortality has been challenged by Waterer et al.¹⁴ By suggesting that time pressures from implementation of a TFAD quality standard of less than 4 hours failed to improve the TFAD, but was responsible for a deterioration in diagnostic accuracy of pneumonia in our ED, our data support their skepticism. Consistent with these concerns, the latest version of guidelines for the management of adult patients with CAP by the Infectious Diseases Society of America/American Thoracic Society¹⁷ has advised that patients receive antibiotics in the ED, rather than specifying a number of hours for the TFAD. The potential morbidity and mortality resulting from an increased number of incorrect diagnoses, inappropriate antibiotic therapy, and delays in establishing the correct diagnosis in patients who do not have CAP must be considered if a TFAD of less than 4 hours is considered to be a core quality measure. We conclude that the lack of prospective trials, the lack of prior consideration of the effect of inaccurate diagnoses of CAP resulting from pressure to decrease TFAD, and the lack of a credible gold standard for the diagnosis of pneumonia cast doubt on the validity of TFAD as a reliable criterion of quality of care.

Correspondence: James A. Welker, DO, Franklin Square Hospital Center, 9000 Franklin Square Dr, Baltimore, MD 21237 (jimwelker@hotmail.com).

Accepted for Publication: August 6, 2007.

Author Contributions: All authors had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Welker, Huston, and McCue. Acquisition of data: Welker, Huston, and McCue. Analysis and interpretation of data: Welker, Huston, and McCue. Drafting of the manuscript: Welker and McCue. Critical revision of the manuscript for important intellectual content: Welker, Huston, and McCue. Statistical analysis: Welker and Huston. Obtained funding: Welker. Administrative, technical, and material support: Welker, Huston, and McCue. Study supervision: Welker, Huston, and McCue.

Financial Disclosure: None reported.

Funding/Support: This study was supported by an unrestricted research grant from Pfizer US Pharmaceuticals.

Role of the Sponsor: The funding body had no role in the design and conduct of the study; in the collection, analysis, and interpretation of the data; or in the preparation, review, or approval of the manuscript.

Additional Contributions: Medical students Katherine Sullivan and Christopher Mitchell spent countless hours participating in the medical record review process; and Norman Dubin, PhD, provided statistical analysis.