Customize your JAMA Network experience by selecting one or more topics from the list below.
O'Leary TJ, Tellado M, Buckner S, Ali IS, Stevens A, Ollayos CW. PAPNET-Assisted Rescreening of Cervical Smears: Cost and Accuracy Compared With a 100% Manual Rescreening Strategy. JAMA. 1998;279(3):235–237. doi:10.1001/jama.279.3.235
From the Department of Cellular Pathology, Armed Forces Institute of Pathology, Washington, DC.
Toward Optimal Laboratory Use section editor: George D. Lundberg,
MD, Editor, JAMA.
Context.— The Food and Drug Administration has recently approved several devices
that use computerized image analysis to rescreen Papanicolaou (Pap) smears
that have already been examined by cytotechnologists. Physicians and laboratories
must decide whether the utility of these devices justifies the cost.
Objective.— To determine the effectiveness and cost of PAPNET-assisted rescreening
in identifying cervical abnormalities not identified by manual rescreening.
Design.— PAPNET-assisted rescreening of 5478 Pap smears obtained in 1994 and
1995 previously identified as "within normal limits" or "benign changes" on
both initial and random screening.
Patients.— Female service members and dependents aged 12 to 88 years.
Setting.— Air Force clinics in the United States and Japan.
Intervention.— Rescreening of Pap smears by PAPNET, followed by reevaluation of abnormal
smears by the consensus panel, consisting of 3 cytotechnologists and 3 pathologists.
Main Outcome Measures.— Proportion of Pap smears initially screened as normal identified as
abnormal by both PAPNET and consensus panel; costs of rescreening.
Results.— PAPNET screening identified 1614 (29%) slides requiring additional microscopic
review. On further review, 448 (8% of total) had possibly abnormal cells.
Ultimately, 11 of these cases were reviewed by the consensus panel for potentially
atypical cells. Of these 11 cases, 5 were reclassified as atypical squamous
cells of undetermined significance (ASCUS) and 1 as atypical glandular cells
of undetermined significance (AGUS). No additional squamous intraepithelial
neoplasia (SIL) was identified in these smears; the patient with a diagnosis
of AGUS on rescreening was diagnosed as having a low-grade SIL (LSIL) on follow-up.
Costs were $5825 to $33781 for each additional ASCUS or AGUS diagnosis. A
cost of $17475 to $101343 is expected for each case of LSIL identified by
PAPNET-assisted rescreening and not by traditional manual rescreening.
Conclusions.— PAPNET-assisted rescreening identified a few more cases of ASCUS than
did manual rescreening, but at a relatively high cost. The costs of rescreening
should be carefully compared with the expected efficacy in reducing cervical
ALTHOUGH the Papanicolaou (Pap) smear has contributed to a significant
decrease in cervical cancer mortality, conventional cytologic examination
misses a significant number of premalignant lesions in which diagnostic cells
are present on the slide. Among the tools that are available for reducing
the false-negative rate are automated rescreening devices that have been approved
recently by the Food and Drug Administration.1-4
The cost-effectiveness of automated rescreening devices is likely to
depend on other laboratory costs, the prevalence of significant lesions within
the population screened, and the false-negative rate of the laboratory. Among
the alternatives to machine-assisted rescreening is manual rescreening of
slides previously diagnosed as negative. The advantage to manual rescreening
is timeliness and possibly lower costs; the disadvantage is a potentially
lower detection rate, particularly if computer-assisted rescreening relies
on features that are not ordinarily appreciated by cytotechnologists.
In this article we report the results of a study that assessed the effectiveness
of a computer-assisted rescreening system (PAPNET, Neuromedical Systems Inc,
Suffern, NY) in identifying cellular abnormalities in Pap smears that had
been previously diagnosed as "within normal limits" or "benign cellular changes"
after both primary screening and a second manual rescreening. The study allows
us to compare the cost and effectiveness of PAPNET-assisted rescreening with
that of manual rescreening, because it allows us to identify cases that were
missed by a second manual screen, to calculate the cost of identifying these
additional cases, and to compare this cost with that associated with completely
The Armed Forces Institute of Pathology receives and screens approximately
40000 cases per year from 8 to 12 Air Force hospitals and clinics located
throughout the United States and Japan. Pap smears are taken from active duty
and retired military personnel, and from eligible wives and daughters. We
identified 5478 cases among those diagnosed in 1994 and 1995 that had been
interpreted as "within normal limits" or "benign cellular changes" on both
primary screening and a 10% random rescreen. These cases were sequential and
included Pap smears from women of all ages (Table 1). Approximately 95% of all cases examined by the laboratory
are diagnosed as within normal limits (87%), infection (3%), or reactive changes
(5%). Approximately 2% of our cases are diagnosed as low- or high-grade squamous
intraepithelial lesion (SIL); from 2% to 3% are diagnosed as atypical squamous
cells of undetermined significance (ASCUS), and the rest are classified as
unsatisfactory. Invasive carcinomas are identified only rarely.
Slides were imaged using the PAPNET system, and the digitized images
of the 128 most "abnormal" regions were reviewed by 1 of 4 individuals (3
cytotechnologists, 1 cytopathologist) who had been trained by Neuromedical
Systems. Slides deemed appropriate for review, based on the Neuromedical Systems
criteria, were manually rescreened for a third time. If the smear was diagnosed
as "atypical squamous/glandular cells of undetermined significance" or more
abnormal, the case was further reviewed by a consensus panel consisting of
3 pathologists and 3 cytotechnologists and a consensus opinion was achieved.
Of the 5478 Pap smears imaged using the PAPNET system, 3864 (71%) were
triaged as negative without further microscopic review. Of the remaining 1614
cases selected for microscopic review, 1166 were identified because no definitive
endocervical component was identified within the PAPNET images. Microscopic
review revealed that 257 (22%) of these cases actually demonstrated endocervical
cells elsewhere on the slide (typically the edge), while 909 did not.
The remaining 448 cases that underwent manual review did so because
128 tiles were not available for review, only a scant squamous component was
present, or uninterpretable or atypical-appearing cells were present on the
PAPNET display. In 11 of these cases, the reviewer believed that previously
undiagnosed abnormal cells might be present, and the case was further reviewed
microscopically by the consensus panel. Of these 11 cases, 5 were classified
as atypical squamous cells of undetermined significance (ASCUS) and 1 as atypical
glandular cells of undetermined significance (AGUS). In no case did observers
universally favor either a reactive or a preneoplastic origin for the cells
giving rise to the diagnosis. In 1 case (that reclassified as AGUS), the patient
has had 2 subsequent Pap smears demonstrating low-grade squamous intraepithelial
neoplasia (LSIL); 2 patients whose smears were reclassified as ASCUS have
demonstrated a single normal Pap smear subsequent to the slide entered into
this investigation. We have not been able to obtain follow-up for the remaining
3 patients; their clinical courses remain unknown.
Manual screening was found to take approximately 3 times longer than
PAPNET-assisted rescreening in those cases in which manual rescreening was
not required (approximately two thirds of cases), yielding an average savings
of 2 minutes per case for PAPNET-assisted rescreening.
The clinical utility of a rescreening procedure depends on both its
efficacy and cost. PAPNET testing is expensive; our study was designed to
determine whether the increased yield of "abnormal" diagnoses obtained using
PAPNET for rescreening justifies this cost. For purposes of this analysis,
we have assumed that all abnormal cases identified by manual rescreening alone
would also be identified using PAPNET-assisted rescreening. Our study demonstrates
that PAPNET-assisted rescreening identifies a small number of abnormal smears
(0.11%) that are not identified on a single manual rescreening. This result
is similar to that reported by Ashfaq et al,5
and is substantially lower than that reported by other investigators.6-15
The differences are readily explained by the study design. Most other studies
have been conducted on samples with a relatively high percentage of abnormal
specimens, such as known false negatives,6,7,13,16
cases originally diagnosed as ASCUS,12 slides
that had been enriched in "abnormal" diagnoses,8
or cases that had not been previously screened.9
Our study was conducted on a sample expected to have a particularly low fraction
of false negatives—slides that had already undergone a negative rescreen.
In contrast to all the above studies, therefore, ours strictly addressed the
question of whether PAPNET-assisted rescreening identified abnormal smears
that had not been diagnosed as abnormal on the basis of a primary screen plus
a manual rescreen.
While the value of detecting high-grade, or even low-grade squamous
intraepithelial lesions is not questioned, the value of ASCUS and AGUS diagnoses
in guiding patient treatment has not been established. For this reason, it
is difficult to conclude on the basis of our results that there is any clinical
value to PAPNET-assisted rescreening. Given the low number of additional abnormal
cases that we identified using the PAPNET system, cost-effectiveness analysis
is difficult. For the following analysis, we assume that the PAPNET-assisted
rescreen might have been expected to identify 2 additional cases of LSIL (which
it did not, but which might be expected on the basis of 6 ASCUS diagnoses
and other reports in the literature).
We assume in this cost analysis that the total cost associated with
manual rescreening is the labor cost for the cytotechnologist (Crescreen). The cost (CPAPNET) for PAPNET-assisted rescreening is
shown in the 2 equations below:
where Csystem consists of the fees charged by Neuromedical
Systems, Inc, for digitization and analysis of the slides, Creview
is the cost for cytotechnologist review of the PAPNET images, and C*rescreen is the cost for rescreening those slides that are identified
as potentially abnormal during the cytotechnologist review of PAPNET images.
where prescreen is the probability that a given slide will
be selected for manual rescreening based on review of the PAPNET images. (Based
on the data presented above, we have assumed prescreen = 0.295
in our analyses.)
A cost analysis for our laboratory has identified the cytotechnologist
cost for either initial screening or manual rescreening (Crescreen)
to be approximately $3 per slide screened, a figure also used by Kaminsky
et al17 in their analysis of rescreening strategies.
We use this cost for completely rescreening a slide, whether as the sole rescreening
strategy or if the slide was identified as potentially abnormal on the basis
of PAPNET images. C*rescreen is thus $0.88. Based on the time analysis
above, cytotechnologist cost for initial PAPNET examination (ie, review of
stored images, or Creview) is one third this cost ($1). The PAPNET
system charges (Csystem) were $7.50 per slide, for a total cost
of PAPNET-assisted rescreening (CPAPNET) of $9.38 per slide. The
marginal cost of PAPNET-assisted rescreening over that of 100% manual rescreening
is $6.38 per slide. Based on our data above, the use of PAPNET will yield
1 more ASCUS or AGUS diagnosis than will manual rescreening for every 913
cases rescreened, at a total additional cost of $5825. The use of PAPNET might
be expected to yield 1 more LSIL diagnosis than will manual rescreening for
every 2739 cases rescreened, at an additional cost of $17475. If the costs
quoted in advertisements for the PAPNET system ($40 per case) are used in
this analysis, the marginal cost of PAPNET over 100% rescreening is 913×$37,
or $33781 per case of ASCUS or AGUS identified, and 2739×$37, or $101343
per expected case of LSIL identified. These marginal costs are somewhat higher
than those estimated by Hutchinson18 for PAPNET-assisted
rescreening, apparently reflecting the fact that PAPNET-assisted rescreening
was less effective in our study than assumed by Hutchinson.
Although Hutchinson's analysis has been criticized for relying on an
inappropriate previously published data set,19
our study nevertheless substantiates the general conclusions regarding the
cost-effectiveness of PAPNET-associated rescreening.
We may compare the costs of this rescreening strategy with the costs
of the routine rescreening program used in our laboratory. Our program consists
of 100% rescreening of patients identified as "high risk"20
because of a history of previously positive Pap smears or because the smears
were obtained in colposcopy clinic, and 10% "random rescreening." Cases that
were identified as abnormal on initial rescreening were subjected to consensus
review of 5 cytotechnologists and pathologists. During 1996, rescreening of
approximately 4970 cases identified 3 LSIL diagnoses, 3 AGUS diagnoses, and
11 ASCUS diagnoses, at a cost per LSIL diagnosis of $4970, and a cost per
ASCUS or AGUS diagnosis of $1065. These costs are also higher than those estimated
by Hutchinson,18 reflecting our identification
by rescreening of a smaller fraction of "missed" cases than assumed in Hutchinson's
Reduction of cervical cancer mortality can be achieved by a number of
different approaches.1 An attractive alternative
to rescreening is to implement more effective screening programs, ie, performing
Pap smears on women who do not currently get them. As noted above, our laboratory
identifies approximately 2 cases of SIL and 2 to 3 cases of ASCUS for every
100 Pap smears examined. Based on an office visit cost of $13521
and our laboratory's charge of $10 per Pap smear, SIL can be identified for
$7250 per case by screening populations similar to ours. Our laboratory's
population consists of women with a relatively high degree of access to care.
Waugh et al22 have shown that a population-based
cervical screening program aimed at women who have not previously had a Pap
smear cost approximately £5780 (US$9826 [approx]) per life saved. Schwarz
et al23 have demonstrated that patients who
have had negative Pap smears reported within the previous 4 years were more
likely to have negative Pap smear results than were women who had never been
screened. We thus infer that strategies aimed at increasing the number of
women screened, particularly among those who have not had a Pap smear, are
likely to be much more cost-effective than computer-assisted rescreening at
reducing cervical cancer morbidity and mortality.
The relative utility of rescreening, in turn, will be strongly influenced
by the prevalence of disease in the screened population, by the sensitivity
of initial screening, and by the cost associated with rescreening.24 Clearly, large reductions in the cost of PAPNET-assisted
rescreening would contribute substantially to its cost-effectiveness. In addition,
cost-effectiveness will be higher for laboratories that screen populations
having a higher prevalence of cervical disease and for laboratories that have
a higher false-negative rate on initial screening.
Since absence of diagnostic cells in the smear is a more frequent cause
of missed cases than is screening error,25
and because additional screening errors may occur because of sample thickness,
air-drying artifacts, and other controllable factors, educational efforts
that teach more effective sampling of the cervix may also be more cost-effective
than any rescreening strategy, whether manual or automated.
In summary, PAPNET-assisted rescreening identifies a few more cases
of ASCUS than does manual rescreening, but at a relatively high cost that
must be carefully considered given its expected efficacy in reducing cervical