Mandl LA, Solomon DH, Smith EL, Lew RA, Katz JN, Shmerling RH. Using Antineutrophil Cytoplasmic Antibody Testing to Diagnose VasculitisCan Test-Ordering Guidelines Improve Diagnostic Accuracy?. Arch Intern Med. 2002;162(13):1509-1514. doi:10.1001/archinte.162.13.1509
Copyright 2002 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.2002
Antineutrophil cytoplasmic antibodies (ANCAs) are strongly associated with Wegener granulomatosis, Churg-Strauss angiitis, microscopic polyangiitis, and pauci-immune glomerulonephritis, referred to collectively as ANCA-associated vasculitis (AAVs). It is unclear how accurate ANCA measurement is for diagnosing AAV in diverse populations or whether proposed ANCA test-ordering guidelines improve test performance.
We assembled a retrospective case series of hospitalized and ambulatory patients from 2 academic medical centers to assess the diagnostic accuracy of ANCA measurement by enzyme-linked immunosorbent assay in identifying cases of AAV. In addition, we assessed the effect of applying proposed ANCA test-ordering guidelines on test performance.
For ANCA testing, sensitivity was 81%; specificity, 98%; positive predictive value, 54%; and negative predictive value, 99%. There were no significant changes in operating characteristics after applying the guideline criteria. Using guidelines would have decreased ANCA test ordering by 23% and would have decreased the false-positive rate by 27%. No cases of AAV would have been missed if only patients fulfilling the guidelines were ANCA tested.
A positive result on an enzyme-linked immunosorbent assay ANCA test, as it is currently ordered, is not a definitive diagnostic indicator of AAV. Compliance with guidelines for ANCA testing would decrease the number of false-positive results and has the potential to reduce total test expenditures.
ANTINEUTROPHIL cytoplasmic antibodies (ANCAs) are strongly associated with Wegener granulomatosis (WG), microscopic polyangiitis (MPA), Churg-Strauss vasculitis, and necrotizing pauci-immune glomerulonephritis.1 These diseases can be collectively referred to as ANCA-associated vasculitis (AAVs). In referral populations with established WG, the sensitivity and specificity of ANCA measurement are reported to be as high as 95%.2,3 As a result, ANCA testing has become part of the standard evaluation for all AAVs.
However, it is unclear how accurate ANCA testing is in diagnosing new cases of AAV. Among patients without a previous diagnosis of WG, the sensitivity may be as low as 34%.4 False-positive ANCA test results have been reported in a number of rheumatologic and nonrheumatologic conditions, including rheumatoid arthritis (RA), human immunodeficiency viral syndrome, monoclonal gammopathy, tuberculosis, and subacute bacterial endocarditis.5- 9 Studies limited to patients seen by rheumatologists or that only include patients with known positive ANCA test results10,11 may not reflect patterns of ANCA test ordering in clinical practice. To avoid spectrum bias and ensure appropriate interpretation of test results, it is important to establish the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) in the patient population in whom ANCA testing is actually used. In addition, older studies have used indirect immunofluorescence (IIF) testing without antigen-specific enzyme-linked immunosorbent assay (ELISA),10 and IIF testing alone is no longer considered the gold standard method of ANCA measurement.12
The likelihood that a positive test result indicates disease is a test's PPV. It depends not only on the test's sensitivity and specificity, but also on disease prevalence. The more common a disease is in the population being tested, the more likely it is that a positive test result is a "true-positive" result. Limiting ANCA testing to patients whose clinical signs and symptoms are most suggestive of AAV could effectively increase the prevalence (or prior probability) of AAV in the group being tested, thus improving ANCA's PPV. This is one rationale behind the development of test-ordering guidelines. Guidelines for ANCA testing have been proposed (Table 1).13 It has been suggested by the guideline authors that if ANCA testing was restricted to patients meeting the guideline criteria, ANCA testing would be "probably 95% sensitive and 90% specific for AAV, with a much higher positive predictive value than when ANCA testing is used in nonselected hospitalized patients."13
There are 2 main techniques used to identify ANCA. Indirect immunofluorescence uses ethanol-fixed neutrophils to demonstrate a perinuclear (p-ANCA)– or cytoplasmic (c-ANCA)–staining pattern. These patterns reflect the fixation and staining of antigenic material, but are not antigen specific. While a c-ANCA pattern is usually, although not always, due to a specific serine proteinase 3, a p-ANCA pattern can result from the presence of a number of different target antigens, such as elastase, cathepsin G, thyroperoxidase, and lactoferrin.14,15 In addition, IIF testing is dependent on the expertise of the laboratory technicians in interpreting the results. To avoid the problems of false-positive results and operator dependence in IIF testing, antigen-specific ELISAs are used to test specifically for myeloperoxidase (MPO) and proteinase 3, the antigens associated with AAV. A positive IIF test result alone is not specific for AAV, and almost all patients with clinically active ANCA-associated small vessel vasculitis, such as the patients in the present study, have high ELISA titers at presentation.13 Some authors suggest that IIF testing be avoided altogether.16 Therefore, in this analysis we considered an ANCA test result to be negative if the ELISA test result was negative, even if the IIF test result was positive.
We conducted a retrospective review of 615 consecutive patients for whom ANCA testing was performed, prior to the establishment of test-ordering guidelines. We assessed how accurately ANCA testing identified new cases of AAV in this population. We also operationalized the proposed guidelines so they could be applied to patients in a reproducible way and determined whether using the guidelines would have improved the test's diagnostic accuracy.
A review of laboratory records identified consecutive patients for whom an ANCA test was ordered between April 1997 and July 1998 at either the Beth Israel Deaconess Medical Center or the Brigham and Women's Hospital, both located in Boston, Mass. Patients were either hospitalized at the time of testing or were seen at a hospital-based or community practice.
Our aim was to determine the accuracy of ANCA testing as a diagnostic test in patients without a known disease process to explain their presentation. Therefore, we excluded patients with established AAV or other vasculitis. We included only those patients whose first ANCA test occurred during the study period. The ANCA test-ordering guidelines proposed by Savige et al13 advocate limiting ANCA testing to clinical indications in which there is "no other obvious cause." We therefore also excluded patients with a previously known malignancy that could explain signs and symptoms such as lung nodules, hemoptysis, or epistaxis.
Each patient's medical record was retrospectively reviewed by 1 of 4 rheumatologists, blinded to ANCA test results. In addition, 20 charts were reviewed by 2 different reviewers, and there was 100% agreement for both diagnosis of AAV and whether the patient satisfied test ordering criteria. If a reviewer had questions regarding a patient's diagnosis or ordering criteria, that patient was presented to all reviewers without revealing his or her ANCA status, and a decision was reached by consensus. We recorded information available prior to the date the ANCA test was ordered using a standardized abstraction form (available on request), which included signs, symptoms, and laboratory values. If multiple ANCA tests were performed on the same patient during the data collection period, we included only the first test in the analysis.
Clinical ANCA test-ordering guidelines were formulated as part of an international consensus statement on ANCA testing and reporting.13 Data collected in the medical record abstraction were used to determine whether patients in this study fulfilled guideline criteria. This was accomplished by creating specific clinical definitions for each guideline criterion (Table 1). If a patient had one of the recommended clinical indications for ANCA testing at the time that the ANCA test was ordered, the patient was considered to have fulfilled criteria for ANCA testing.
We determined whether patients undergoing ANCA testing fulfilled criteria for one of the following AAVs: WG, Churg-Strauss angiitis, MPA, or pauci-immune glomerulonephritis. All cases of WG or Churg-Strauss angiitis satisfied the American College of Rheumatology criteria.17,18 If no chest radiograph was performed, we considered radiographic evidence of erosive sinusitis as a disease criterion. Histologic evidence of granulomatous inflammation was used as 1 criterion for WG, but alone was not sufficient for the diagnosis. The definition of MPA was based on the Chapel Hill Consensus nomenclature, as demonstrated by a biopsy result showing necrotizing, small vessel vasculitis with few or no immune deposits and clinical features of pulmonary inflammation.19 If a patient satisfied criteria for both MPA and WG simultaneously, those with upper respiratory symptoms (eg, sinusitis, epistaxis, or ear pain) or nodules on chest radiographs were considered to have WG, and those with lower pulmonary symptoms (eg, hemoptysis) were considered to have MPA. Patients with pauci-immune glomerulonephritis had a biopsy result showing pauci-immune glomerulonephritis and no evidence of pulmonary involvement. This was for classification purposes because histologically, these patients cannot be distinguished from patients with MPA limited to the kidneys. If patients did not satisfy one of these disease criteria, biopsy results showing necrotizing vasculitis were not considered to be diagnostic of an AAV. Patients assigned a diagnosis of AAV by their physician without fulfilling criteria were also not considered to have AAV in this analysis. Diagnosis of a new AAV was made at the time of ANCA testing. Records were reviewed 8 to 12 months after the index ANCA test was ordered, and no patient was subsequently diagnosed as having an AAV during that period. Other medical diagnoses were established by each patient's treating physician.
All ANCA tests ordered at Beth Israel Deaconess Medical Center were performed at the Massachusetts General Hospital Laboratory in Boston (laboratory 1). Tests ordered at Brigham and Women's Hospital were performed at either laboratory 1 or Quest Diagnostic Laboratory in Capistrano, Calif (laboratory 2). At laboratory 1, IIF testing and direct antigen-specific ELISA were performed on each sample, as described elsewhere.20 At laboratory 2, only ELISA testing was performed. If no titer was reported, or if results were recorded only as "positive" in the hospital records, the testing laboratories were contacted directly to obtain more specific data. Each laboratory determined its own threshold for a positive test. In this study we considered an ANCA test result to be positive if the laboratory reported a positive anti-MPO or anti–proteinase 3 antibody on ELISA testing. Borderline results, as defined and reported by each laboratory, were considered negative. Positive p-ANCAs or c-ANCAs without an associated positive anti-MPO or anti–proteinase 3 titer were considered negative in our analysis.
Sensitivity, specificity, PPV, and NPV were calculated using standard contingency tables (see the "Results" section).21 Comparisons between groups were analyzed by the Fisher exact test for categorical variables using a 2-tailed significance level of P<.05. All data were analyzed using SAS software (SAS Institute Inc, Cary, NC). The 95% confidence intervals (CIs) for specificity, sensitivity, PPV, and NPV were determined using methods described by Fleiss.22 Analysis was also performed without excluding patients with systemic malignancies, and there was no significant difference in ANCA test operating characteristics (ie, sensitivity, specificity, NPV, and PPV). Only the results excluding systemic malignancies are presented.
Of the 615 patients who had an ANCA test ordered during the study period, we excluded 118 (21 medical records were unobtainable, 50 patients had a previous diagnosis of vasculitis, 26 patients had a systemic malignancy, and 21 patients had previous ANCA testing prior to our data collection period). The characteristics of the 497 remaining patients are given in Table 2. Sixteen patients (3%) were diagnosed as having AAV (10 had WG, 4 had MPA, and 2 had necrotizing pauci-immune glomerulonephritis) (Table 3). Each of these patients fulfilled at least 1 criterion for ordering an ANCA test. The most common clinical signs and symptoms among all patients at the time the ANCA test was ordered were dyspnea (26%), fever (24%), renal failure (22%), and cough (21%).
We first calculated the ANCA test operating characteristics for all 497 patients:
The sensitivity of ANCA testing was 81% (95% CI, 62%-100%); specificity, 98% (95% CI, 96%-99%); NPV, 99% (95% CI, 99%-100%); and PPV, 54% (95% CI, 34%-74%).
Operating characteristics were then calculated for the 381 patients fulfilling at least 1 guideline criterion for ordering an ANCA test:
The sensitivity of ANCA testing for this subset of patients was 81% (95% CI, 62%-100%); specificity, 98% (95% CI, 96%-99%); NPV, 99% (95% CI, 98%-100%), and PPV, 62% (95% CI, 41%-83%) with the following results.
There were no significant differences between the 2 groups. However, there was a trend toward better test performance among inpatients (PPV, 69%) compared with outpatients (PPV, 36%) (P = .2). Sensitivity was 100% among inpatients and 57% among outpatients (P = .06). If testing had been limited to patients satisfying guideline criteria, the overall number of ANCA tests would have been reduced by 116 (23%), eliminating 3 of 11 false-positive results.
In our population the false-positive rate was 2.2%, with 10 of 11 of the false-positive results being at least twice the upper limit of the normal range for antigen-specific ELISAs. Most false-positive results were anti-MPO antibodies (Table 4). Four of 11 false-positive results were observed in patients with RA. The false-positive rate in those patients meeting at least 1 guideline criterion was 2.1%. Of 15 patients with either sarcoidosis, lethal midline granuloma, or tuberculosis (granulomatous disorders that may mimic AAV), none had positive ANCA test results.
This study evaluated the performance characteristics of ANCA testing when used to identify AAV in patients without a previous diagnosis of vasculitis. Unlike other studies that looked only at rheumatology patients being evaluated for vasculitis, we examined all patients on whom ANCA testing was requested by practicing clinicians, better reflecting how ANCA testing is actually used.4 While ANCA testing has been promoted as a potentially important diagnostic tool ever since its initial description in 1982,1 in the population we studied, 11 of 24 patients with a positive ANCA test results were not diagnosed as having AAV, suggesting a false-positive rate of 46%. Importantly, even though all patients had clinically symptomatic disease, 3 (19%) of 16 patients with AAV had negative ANCA test results. In our study, the PPV of ANCA testing (54%) was too low to be considered a definitive diagnostic test. On the other hand, ANCA testing did demonstrate high specificity (98%) and NPV (99%). Given the rarity of AAV, even among patients for whom ANCA testing is requested, a high NPV may have little clinical impact; in the present study, the chances of having an AAV fell from 3% to 1% when the ANCA test result was negative.
Of the 11 false-positive results, 4 (36%) occurred in patients with RA, all of whom were tested at laboratory 1. Up to 36% of patients with RA may have an associated p-ANCA or atypical ANCA pattern by IIF, which are not specific for AAV. In addition, some studies have shown that up to 10% of patients with RA demonstrate anti-MPO antibodies by ELISA testing.9,23 However, these anti-MPO antibodies may be due to cross-reactions with unscreened antigens such as lysozymes or lactoferrin, which can occur when commercial kits are used for ANCA testing.24 Laboratory 1 does not use a commercial kit and has extensive experience with ANCA testing. Published data from this laboratory show that in their laboratory, only 1.4% of RA cases are associated with anti-MPO antibodies.25 Because all our anti-MPO–positive RA patients were tested at laboratory 1, we do not believe these results were due to contamination with unscreened antigens. Even though RA is only rarely associated with anti-MPO antibodies, our data suggest that RA may comprise a disproportionately large percentage of false-positive results.
The diagnostic accuracy of ANCA testing in our population mirrors the experience with other tests, such as the carcinoembryonic antigen marker for colon cancer26 or the IIF assay for Lyme disease.27 Although initially thought to be excellent tools for diagnosis, both tests proved to be imprecise indicators of disease when applied widely to heterogeneous populations. As with ANCA testing, these tests were developed in populations with high disease prevalence, but were often applied clinically in settings with low rates of disease. Selective testing based on clinical settings of high disease likelihood will yield more accurate test results. However, our study results suggest that even among patients who satisfy guideline criteria, the PPV is only 62%. This is not high enough for ANCA testing to supplant a confirmatory tissue biopsy, which is the generally accepted gold standard in patients among whom other causes of secondary vasculitis have been ruled out. The high specificity of ANCA testing and high NPV noted in our study is also tempered by the recognition that other vasculitides, such as hypersensitivity vasculitis, polyarteritis nodosa, and temporal arteritis, are typically ANCA negative. Therefore, a negative ANCA test result cannot be used to rule out vasculitis in general.
To our knowledge, the proposed clinical indications for ANCA testing have never been validated.13 In our study, following ANCA test-ordering guidelines would not have led to any missed cases of AAV, since all patients with AAV satisfied at least 1 of the guideline criteria. If ANCA testing had only been ordered for patients meeting at least 1 guideline criterion, the number of tests would have decreased by 23%, reducing direct laboratory charges by $24 738 (1999 dollars). This is likely a conservative estimate of cost savings because it does not include any subsequent downstream costs. In addition, application of the guidelines would have prevented 27% of patients who had false-positive results from being tested. Decreasing the number of false-positive results could prevent further invasive procedures or harmful adverse effects from inappropriate treatment. These data suggest that limiting ANCA test ordering according to the proposed guidelines could result in cost savings without compromising patient care.
This study has a number of limitations. Because the authors of the ANCA test-ordering guidelines did not state how an individual patient would fulfill each criterion, we operationalized the criteria to reflect how clinicians might use them in typical clinical situations. Our method of defining the guidelines may have influenced our results. Blood samples were not all analyzed at the same laboratory, and laboratory 2 performed only ELISA testing. However, the false-positive rates were similarly low (2% vs 3%) at both laboratories, suggesting similar test performance at both centers. This study was performed at 2 large urban tertiary care centers, possibly limiting generalizability. Only 10% of patients were in the intensive care unit, however, and 50% of the ANCA tests were performed on outpatients from primary care and specialty practices, assuring that a wide spectrum of patients were represented. Secular changes in management, such as relying on ANCA test results for diagnosis, may have dissuaded physicians from obtaining tissue biopsy specimens. This may have precluded some patients with AAV from satisfying our definitions of disease and thus contributed to misclassification bias. Milder or evolving cases of AAV may be missed by the American College of Rheumatology17,18 and Chapel Hill Consensus19 disease definitions, also leading to misclassification bias. This seems unlikely because all patients considered free of AAV remained so after 8 to 12 months of follow-up.
It has been argued that ANCA testing may be most effective when used to separate AAV patients from non-AAV patients with pulmonary or pulmonary-renal syndromes, rather than as a diagnostic test for AAV.28 While this may be true, in many clinical settings, testing for ANCA has become part of the routine investigation of patients with suspected vasculitis. It is therefore important to determine how ANCA testing performs in this context. In addition, when guidelines are proposed, they should be validated in clinical practice. The use of ANCA test-ordering guidelines would have decreased the number of tests in this population by 23% without missing any cases of AAV. Thus, our data underscore that the ANCA test, as currently ordered, is not a definitive diagnostic test for AAV and that proposed guidelines are appropriate and could reduce unnecessary testing as well as false-positive results.
Accepted for publication October 11, 2001.
Drs Mandl and Katz are supported by grant P01 AR36308 from the National Institutes of Health, Bethesda, MD. Dr Katz is also supported by grant 1R01 HS09775-01 from the Agency for Healthcare Research and Quality, Rockville, Md, and a clinical science grant from the Arthritis Foundation, Atlanta, Ga. Dr Mandl is the recipient of a Physician Scientist Award from the Arthritis Foundation. Dr Solomon is supported by an Investigator Award from the Arthritis Foundation.
A portion of these data were presented at the American College of Rheumatology Meeting, Boston, Mass, November 15, 1999.