[Skip to Content]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address 54.211.168.204. Please contact the publisher to request reinstatement.
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
Download PDF
Image description not available.
Figure 1.—Result of search strategy. The asterisk indicates that 1 study76 presented criteria for 2 separate tests.
Image description not available.
Figure 2.—Paradigm for the classification of test appropriateness criteria. Appropriateness criteria could directly compare laboratory tests based on their operating characteristics. Criteria determining the appropriateness of a single test would depend on data based on physiology (eg, knowledge of disease processes), pharmacology (eg, pharmacodynamic tenets), and clinical probability (eg, identifying scenarios where test results would never be useful).
Table 1.—Studies Using Implicit Criteria for Inappropriate Laboratory Use
Image description not available.
Table 2.—Studies Using Explicit Criteria for Inappropriate Laboratory Use for General and Microbiological Tests
Image description not available.
Table 3.—Studies Using Explicit Criteria for Inappropriate Laboratory Use of Cardiac Enzymes and Thyroid Tests
Image description not available.
Table 4.—Studies Using Explicit Criteria for Inappropriate Laboratory Use of Therapeutic Drug Monitoring
Image description not available.
1.
Showstack JA, Schroeder SA, Matsumato MF. Changes in use of medical technologies, 1972-1977: a study of 10 inpatient diagnoses.  N Engl J Med.1982;306:706-712.
2.
Fowkes FG. Containing the use of diagnostic tests.  Br Med J (Clin Res Ed).1985;290:488.
3.
Danzon PM, Manning WG, Marquis MS. Factors affecting laboratory test use and prices.  Health Care Financ Rev.1984:5;23-32.
4.
 Salient Features of the Laboratory Industry in Ontario, 1992-93 . Toronto: Ontario Ministry of Health; 1995.
5.
 Laboratory Services Review—Report to the Ministry of Health . Toronto: Ontario Ministry of Health; 1994.
6.
Epstein AM, McNeil BJ. Relationship of beliefs and behavior in test ordering.  Am J Med.1986;80:865-870.
7.
Williams SV, Eisenberg JM, Pascale LA, Kitz DS. Physicians' perceptions about unnecessary diagnostic testing.  Inquiry.1982;19:363-370.
8.
McGillivray DL, Roberts-Brauer R, Kramer MS. Diagnostic test ordering in the evaluation of febrile children: physician and environmental factors.  AJDC.1993;147:870-874.
9.
Griffith III CH, Wilson JF, Desai NS, Rich EC. Does pediatric housestaff experience influence tests ordered for infants in the neonatal intensive care unit?  Crit Care Med.1997;25:704-709.
10.
Bugter-Maessen AM, Winkens RA, Grol RP.  et al.  Factors predicting differences among general practitioners in test ordering behaviour and in the response to feedback on test requests.  Fam Pract.1996;13:254-258.
11.
Hartley RM, Epstein AM, Harris CM, McNeil BJ. Differences in ambulatory test ordering in England and America: role of doctors' beliefs and attitudes.  Am J Med.1987;82:513-517.
12.
Perraro F, Rossi P, Liva C.  et al.  Inappropriate emergency test ordering in a general hospital: preliminary reports.  Qual Assur Health Care.1992;4:77-81.
13.
Bareford D, Hayling A. Inappropriate use of laboratory services: long term combined approach to modify request patterns.  BMJ.1990;301:1305-1307.
14.
Beck JR. Does feedback reduce inappropriate test ordering?  Arch Pathol Lab Med.1993;117:33-34.
15.
Mozes B, Lubin D, Modan B, Ben-Bassat I, Gitel SN, Halkin H. Evaluation of an intervention aimed at reducing inappropriate use of preoperative blood coagulation tests.  Arch Intern Med.1989;149:1836-1838.
16.
Dans PE, Cafferty L, Otter SE, Johnson RJ. Inappropriate use of the cerebrospinal fluid Venereal Disease Research Laboratory (VDRL) test to exclude neurosyphilis.  Ann Intern Med.1986;104:86-89.
17.
Siegel DL, Edelstein PH, Nachamkin I. Inappropriate testing for diarrheal diseases in the hospital.  JAMA.1990;263:979-982.
18.
Hyams KC. Inappropriate urine cultures in hospitalized patients receiving antibiotic therapy.  Arch Intern Med.1987;147:48-49.
19.
Bush TM, Shlotzhauer TL, Grove W. Serum complements: inappropriate use in patients with suspected rheumatic disease.  Arch Intern Med.1993;153:2363-2366.
20.
Wu AH. Reducing the inappropriate utilization of clinical laboratory tests.  Conn Med.1997;61:15-21.
21.
Rang M. The Ulysses syndrome.  Can Med Assoc J.1972;106:122-123.
22.
Moloney TW, Rogers DE. Medical technology: a different view of the contentious debate over costs.  N Engl J Med.1979;301:1413-1419.
23.
Fineberg HV, Pearlman LA. Low-cost medical practices.  Annu Rev Public Health.1982;3:225-248.
24.
Thamer M, Fitzpatrick KJ, Perry S. Medical and surgical supply costs: case study of a costly low-cost technology.  Int J Technol Assess Health Care.1997;13:526-536.
25.
Lewis S, Foreman J. Low-cost diagnostic technologies and clinical outcomes: the impact of inappropriate utilization.  Int J Technol Assess Health Care.1997;13:501-511.
26.
Reid MC, Lachs MS, Feinstein AR. Use of methodological standards in diagnostic test research: getting better but still not good.  JAMA.1995;274:645-651.
27.
Naylor CD, Guyatt GH.for the Evidence-Based Medicine Working Group.  Users' guides to the medical literature, XI: how to use an article about a clinical utilization review.  JAMA.1996;275:1435-1439.
28.
Clough JD, Calabrese LH, Valenzuela R. The ANA profile: quality and cost-effective laboratory utilization.  Cleve Clin J Med.1989;56:245-248.
29.
Valenstein P, Pfaller M, Yungbluth M. The use and abuse of routine stool microbiology: a College of American Pathologists Q-probes study of 601 institutions.  Arch Pathol Lab Med.1996;120:206-211.
30.
Chitkara YK, McCasland KA, Kenefic L. Development and implementation of cost-effective guidelines in the laboratory investigation of diarrhea in a community hospital.  Arch Intern Med.1996;156:1445-1448.
31.
Murray PR, Washington JA. Microscopic and bacteriologic analysis of expectorated sputum.  Mayo Clin Proc.1975;50:339-344.
32.
Crowson TW, Rich EC, Woolfrey BF, Connelly DP. Overutilization of cultures of CSF for mycobacteria.  JAMA.1984;251:70-72.
33.
Caldwell G, Kellett HA, Gow SM.  et al.  A new strategy for thyroid function testing.  Lancet.1985;1:1117-1119.
34.
Brady MT, Pacini DL, Budde CT, Connell MJ. Diagnostic studies of nosocomial diarrhea in children: assessing their use and value.  Am J Infect Control.1989;17:77-82.
35.
John R, Henley R, Lloyd G, Elder GH. Evaluation of a new strategy for detection of thyroid dysfunction in the routine laboratory.  Clin Chem.1988;34:1110-1114.
36.
Ibrahim H, McKenna MJ, Feldkamp CS. A thyroid testing algorithm: results of a pilot study.  Henry Ford Hosp Med J.1991;39:30-34.
37.
Hayward RA, Shapiro MF, Oye RK. Laboratory testing on cerebrospinal fluid: a reappraisal.  Lancet.1987;1:1-4.
38.
Church DL, Cadrain G, Kabani A, Jadavji T, Trevenen C. Practice guidelines for ordering stool cultures in a pediatric population.  Am J Clin Pathol.1995;103:149-153.
39.
Barbut F, Leluan P, Antoniotti G, Collignon A, Sedallian A, Petit JC. Value of routine stool cultures in hospitalized patients with diarrhea.  Eur J Clin Microbiol Infect Dis.1995;14:346-349.
40.
Asnis DS, Bresciani A, Ryan M, McArdle P, Mollura JL, Ilardi CF. Cost-effective approach to evaluation of diarrheal illness in hospitals.  J Clin Microbiol.1993;31:1675.
41.
Perkins MD, Mirrett S, Reller LB. Rapid bacterial antigen detection is not clinically useful.  J Clin Microbiol.1995;33:1486-1491.
42.
Martin TG, Elenbaas RM, Pingleton SH. Use of peak expiratory flow rates to eliminate unnecessary arterial blood gases in acute asthma.  Ann Emerg Med.1982;11:70-73.
43.
Wenz B, Lampasso JA. Eliminating unnecessary urine microscopy: results and performance characteristics of an algorithm based on chemical reagent strip testing.  Am J Clin Pathol.1989;92:78-81.
44.
Valenstein PN, Koepke JA. Unnecessary microscopy in routine urinalysis.  Am J Clin Pathol.1984;82:444-448.
45.
Rodewald LE, Woodin KA, Szilagyi PG, Arvan DA, Raubertas RF, Powell KR. Relevance of common tests of cerebrospinal fluid in screening for bacterial meningitis.  J Pediatr.1991;119:363-369.
46.
Earle C, Hebert PC. A reader's guide to the evaluation of screening studies.  Postgrad Med J.1996;72:77-83.
47.
Sackett DL, Haynes RB, Guyatt GH.  et al.  Clinical Epidemiology: A Basic Science for Clinical Medicine . 2nd ed. Boston, Mass: Little Brown & Co Inc; 1991:153-172.
48.
Sox Jr HC. Common Diagnostic Tests: Use and Interpretation . Philadelphia, Pa: American College of Physicians; 1987.
49.
Speicher CE. The Right Test: A Physician's Guide to Laboratory Medicine . 2nd ed. Philadelphia, Pa: WB Saunders Co; 1993.
50.
Panzer RJ, Black ER, Griner PF. Diagnostic Strategies For Common Medical Problems . Philadelphia, Pa: American College of Physicians; 1991.
51.
Turnbull TL, Vanden Hoek TL, Howes DS, Eisner RF. Utility of laboratory studies in the emergency department patient with a new-onset seizure.  Ann Emerg Med.1990;19:373-377.
52.
Wenz B, Gennis P, Canova C, Burns ER. The clinical utility of the leukocyte differential in emergency medicine.  Am J Clin Pathol.1986;86:298-303.
53.
Grivell AR, Forgie HJ, Fraser CG, Berry MN. League tables of biochemical laboratory costs: an attempt to modify requesting patterns.  Med J Aust.1982;2:326-328.
54.
Durand-Zaleski I, Rymer JC, Roudot-Thoraval F, Revuz J, Rosa J. Reducing unnecessary laboratory use with new test request form: example of tumour markers.  Lancet.1993;342:150-153.
55.
Tierney WM, McDonald CJ, Martin DK, Rogers MP. Computerized display of past test results: effect on outpatient testing.  Ann Intern Med.1987;107:569-574.
56.
Roberts DE, Bell DD, Ostryzniuk T.  et al.  Eliminating needless testing in intensive care: an information-based team management approach.  Crit Care Med.1993;21:1452-1458.
57.
Ohkubo A. Reducing unnecessary laboratory use through education and regulation.  Rinsho Byori.1995;43:647-652.
58.
Wachtel TJ, O'Sullivan P. Practice guidelines to reduce testing in the hospital.  J Gen Intern Med.1990;5:335-341.
59.
Nightingale PG, Peters M, Mutimer D, Neuberger JM. Effects of a computerised protocol management system on ordering of clinical tests.  Qual Health Care.1994;3:23-28.
60.
Durand-Zaleski I, Rymer JC, Rosa J, Revuz J. Reducing unnecessary laboratory use with a new test request form: the example of tumor markers. In: Abstracts of the Annual Meeting, International Society of Technology Assessment in Health Care . Vol 9. New York, NY: International Society of Technology Assessment in Health Care; 1997:181.
61.
Tierney WM, McDonald CJ, Hui SL, Martin DK. Computer predictions of abnormal test results: effects on outpatient testing.  JAMA.1988;259:1194-1198.
62.
Tierney WM, Miller ME, McDonald CJ. The effect on test ordering of informing physicians of the charges for outpatient diagnostic tests.  N Engl J Med.1990;322:1499-1504.
63.
Tierney WM, Miller ME, Overhage JM, McDonald CJ. Physician inpatient order writing on microcomputer workstations: effects on resource utilization.  JAMA.1993;269:379-383.
64.
Liptzin BA, Williams JS. Laboratory utilization on a university surgical service.  Surgery.1972;71:247-253.
65.
Daniels M, Schroeder SA. Variation among physicians in use of laboratory tests, II: relation to clinical productivity and outcomes of care.  Med Care.1977;15:482-487.
66.
Ley C, Le C, Olshen EM, Reingold AL. The use of serologic tests for Lyme disease in a prepaid health plan in California.  JAMA.1994;271:460-463.
67.
Jackson LM, Shanahan F, Cryan B, Bredin CP, Cronin CC. Appropriateness of laboratory tests: requests for atypical pneumonia serology in a teaching hospital.  Ir J Med Sci.1996;165:93-94.
68.
McLeish WA. Third leukocyte differential counts: utilization study in a general hospital.  Can Med Assoc J.1985;132:616-617.
69.
Seah CS. Unnecessary investigation in medical practice.  Singapore Med J.1983;24:243-247.
70.
Joly JG. Budgetary impact of overutilization of multianalyzer tests.  Union Med Can.1981;110:989-994.
71.
Lepage R, Lessard F, Bannon P, Gauthier Y, Miron M, Joly JG. Over-utilization of tests: the responsibility of the laboratory.  Union Med Can.1984;113:526-532.
72.
Vlasses PH, DiPiro CA, Chalupa D, Hopkins LE. Appropriateness of sampling time for selected serum drug assays.  Hosp Pharm.1982;17:371-373.
73.
Barth JH, Page MD, Bailey IR. An audit of dynamic function tests: errors in the timing of blood specimens.  Clin Endocrinol (Oxf).1993;39:97-100.
74.
Pitterle ME, Sorkness CA, Wiederholt JB. Use of serum drug concentrations in outpatient clinics.  Am J Hosp Pharm.1985;42:1547-1552.
75.
Gotz VP, Lopez LM, Reeves SE, Morrow ME. Clinical utilization of serum theophylline concentrations in a university-affiliated hospital.  Drug Intell Clin Pharm.1982;16:872-874.
76.
Eisenberg JM, Williams SV, Garner L, Viale R, Smits H. Computer-based audit to detect and correct overutilization of laboratory tests.  Med Care.1977;15:915-921.
77.
Gama R, Nightingale PG, Ratcliffe JG. Effect of educational feedback on clinicians' requesting of cardiac enzymes.  Ann Clin Biochem.1992;29(pt 2):224-225.
78.
McConnell TS, Berger PR, Dayton HH, Umland BE, Skipper BE. Professional review of laboratory utilization.  Hum Pathol.1982;13:399-403.
79.
Sandler G. Do emergency tests help in the management of acute medical admissions?  Br Med J (Clin Res Ed).1984;289:973-977.
80.
Bold AM, Corrin B. Use and abuse of clinical chemistry in surgery.  Br Med J.1965;5469:1051-1052.
81.
Boon-Falleur L, Sokal E, Nightingale PG.  et al.  Utilization of laboratory resources: developments in knowledge-based ordering systems.  Int J Biomed Comput.1995;40:17-30.
82.
Branger PJ, Van Oers RJ, van der Wouden JC, van der Lei J. Laboratory services utilization: a survey of repeat investigations in ambulatory care.  Neth J Med.1995;47:208-213.
83.
Bowman RA, Bowman JM, Arrow SA, Riley TV. Selective criteria for the microbiological examination of faecal specimens.  J Clin Pathol.1992;45:838-839.
84.
Pearce GA, Day RO. Compliance with criteria necessary for effective drug concentration monitoring.  Ther Drug Monit.1990;12:250-257.
85.
Penglis S, Martin ED. Evaluation of serum drug level monitoring on a general medical ward.  Aust J Hosp Pharm.1984;14:138-141.
86.
Clague HW, Twum-Barima Y, Carruthers SG. An audit of requests for therapeutic drug monitoring of digoxin: problems and pitfalls.  Ther Drug Monit.1983;5:249-254.
87.
Wing DS, Duff HJ. The impact of a therapeutic drug monitoring program for phenytoin.  Ther Drug Monit.1989;11:32-37.
88.
Levine M, McCollom R, Chang T, Orr J. Evaluation of serum phenytoin monitoring in an acute care setting.  Ther Drug Monit.1988;10:50-57.
89.
Rafeh N, el-Tobgi D. Clinical practice and the use of laboratory tests at the May 15 Hospital in Egypt.  Int J Qual Health Care.1995;7:25-30.
90.
Ruangkanchanasetr S. Laboratory investigation utilization in pediatric out-patient department, Ramathibodi Hospital.  J Med Assoc Thai.1993;76(suppl 2):194-208.
91.
Myers LP, Chapman SA, McPhee SJ, Schroeder SA, Leong JK, Martin AR. Which hospital services are most overused? results from a medical audit.  West J Med.1985;143:397-398.
92.
Dickinson JC. From process to policy: a generic prescription for test over-utilization in the emergency department.  Fam Pract Res J.1987;7:12-21.
93.
Young DW. An aid to reducing unnecessary investigations.  BMJ.1980;281:1610-1611.
94.
Dixon RH, Laszlo J. Utilization of clinical chemistry services by medical house staff: an analysis.  Arch Intern Med.1974;134:1064-1067.
95.
Baigelman W, Bellin SJ, Cupples LA, Dombrowski D, Coldiron J. Overutilization of serum electrolyte determinations in critical care units: savings may be more apparent than real but what is real is of increasing importance.  Intensive Care Med.1985;11:304-308.
96.
Gortmaker SL, Bickford AF, Mathewson HO, Dumbaugh K, Tirrell PC. A successful experiment to reduce unnecessary laboratory use in a community hospital.  Med Care.1988;26:631-642.
97.
Bloomgarden Z, Sidel VW. Evaluation of utilization of laboratory tests in a hospital emergency room.  Am J Public Health.1980;70:525-528.
98.
Albright Jr RE, Christenson RH, Emlet JL.  et al.  Issues in cerebrospinal fluid management: CSF Venereal Disease Research Laboratory testing.  Am J Clin Pathol.1991;95:397-401.
99.
Morris AJ, Wilson ML, Reller LB. Application of rejection criteria for stool ovum and parasite examinations.  J Clin Microbiol.1992;30:3213-3216.
100.
Rhyne RL, Gehlbach SH. Effects of an educational feedback strategy on physician utilization of thyroid function panels.  J Fam Pract.1979;8:1003-1007.
101.
Schectman JM, Elinsky EG, Pawlson LG. Effect of education and feedback on thyroid function testing strategies of primary care clinicians.  Arch Intern Med.1991;151:2163-2166.
102.
Greenlaw CW, Geiger GS, Haug III MT. Use of digoxin serum assays in a nonteaching hospital.  Am J Hosp Pharm.1980;37:1466-1470.
103.
Bussey HI, Hoffman EW. A prospective evaluation of therapeutic drug monitoring.  Ther Drug Monit.1983;5:245-248.
104.
Greenlaw CW, Blough SS, Haugen RK. Aminoglycoside serum assays restricted through a pharmacy program.  Am J Hosp Pharm.1979;36:1080-1083.
105.
Valenstein P, Leiken A, Lehmann C. Test-ordering by multiple physicians increases unnecessary laboratory examinations.  Arch Pathol Lab Med.1988;112:238-241.
106.
Bartlett RC, Mazens-Sullivan M, Tetreault JZ, Lobel S, Nivard J. Evolving approaches to management of quality in clinical microbiology.  Clin Microbiol Rev.1994;7:55-88.
107.
Studnicki J, Bradham DD, Marshburn J, Foulis PR, Straumfjord JV. A feedback system for reducing excessive laboratory tests.  Arch Pathol Lab Med.1993;117:35-39.
108.
Saxena S, Anderson DW, Kaufman RL, Hannah JA, Wong ET. Quality assurance study of cardiac isoenzyme utilization in a large teaching hospital.  Arch Pathol Lab Med.1993;117:180-183.
109.
Schoenenberger RA, Tanasijevic MJ, Jha A, Bates DW. Appropriateness of antiepileptic drug level monitoring.  JAMA.1995;274:1622-1626.
110.
Slaughter RL, Schneider PJ, Visconti JA. Appropriateness of the use of serum digoxin and digitoxin assays.  Am J Hosp Pharm.1978;35:1376-1379.
111.
Guernsey BG, Ingrim NB, Hokanson JA.  et al.  A utilization review of theophylline assays: sampling patterns and use.  Drug Intell Clin Pharm.1984;18:906-912.
112.
Sargenti C, Zelman L, Beauclair T, Garrard E, Boehm R. Evaluation of appropriateness and interpretation of serum theophylline assays.  Drug Intell Clin Pharm.1985;19:380-384.
113.
Ives TJ, Parry JL, Gwyther RE. Serum drug level utilization review in a family medicine residency program.  J Fam Pract.1984;19:507-512.
114.
Finn Jr AF, Valenstein PN, Burke MD. Alteration of physicians' orders by nonphysicians.  JAMA.1988;259:2549-2552.
115.
Erban SB, Kinman JL, Schwartz JS. Routine use of the prothrombin and partial thromboplastin times.  JAMA.1989;262:2428-2432.
116.
Kroenke K, Hanley JF, Copley JB.  et al.  Improving house staff ordering of three common laboratory tests: reductions in test ordering need not result in underutilization.  Med Care.1987;25:928-935.
117.
Goldberg GA, Abbott JA. Explicit criteria for use of laboratory tests.  Ann Intern Med.1974;81:857-858.
118.
Levin B, Cohen SS, Birmingham PH. Effect of pharmacist intervention on the use of serum drug assays.  Am J Hosp Pharm.1981;38:845-851.
119.
Schifman RB, Strand CL, Braun E, Louis-Charles A, Spark RP, Fried ML. Solitary blood cultures as a quality assurance indicator.  Qual Assur Utilization Rev.1991;6:132-137.
120.
Gross PA, Van Antwerpen CL, Hess WA, Reilly KA. Use and abuse of blood cultures: program to limit use.  Am J Infect Control.1988;16:114-117.
121.
Flynn TW, Pevonka MP, Yost RL, Weber CE, Stewart RB. Use of serum gentamicin levels in hospitalized patients.  Am J Hosp Pharm.1978;35:806-808.
122.
Dolan JG, Bordley DR, Mushlin AI. An evaluation of clinicians' subjective prior probability estimates.  Med Decis Making.1986;6:216-223.
123.
Nightingale SD. Risk preference and admitting rates of emergency room physicians.  Med Care.1988;26:84-87.
124.
Nightingale SD, Grant M. Risk preference and decision making in critical care situations.  Chest.1988;93:684-687.
125.
Brook RH, Chassin MR, Fink A, Solomon DH, Kosecoff J, Park RE. A method for the detailed assessment of the appropriateness of medical technologies.  Int J Technol Assess Health Care.1986;2:53-63.
126.
Kravitz RL, Laouri M, Kahan JP.  et al.  Validity of criteria used for detecting underuse of coronary revascularization.  JAMA.1995;274:632-638.
127.
Gibb I, Cowan JC, Parnham AJ, Thomas TH. Use and misuse of a digoxin assay service.  BMJ.1986;293:678-680.
128.
Lyon AW, Greenway DC, Hindmarsh JT. A strategy to promote rational clinical chemistry test utilization.  Am J Clin Pathol.1995;103:718-724.
129.
Leese B. Is There Too Much Laboratory Testing?  York, England: University of York; 1991:79.
130.
Fan K, Morris AJ, Reller LB. Application of rejection criteria for stool cultures for bacterial enteric pathogens.  J Clin Microbiol.1993;31:2233-2235.
131.
Wells PS, Hirsh J, Anderson DR.  et al.  Accuracy of clinical assessment of deep-vein thrombosis.  Lancet.1995;345:1326-1330.
132.
Stiell IG, Wells G, Laupacis A.  et al.  Multicentre trial to introduce the Ottawa ankle rules for use of radiography in acute ankle injuries.  BMJ.1995;311:594-597.
133.
Axt-Adam P, van der Wouden JC, van der Does E. Influencing behavior of physicians ordering laboratory tests: a literature study.  Med Care.1993;31:784-794.
134.
Menon D, Hommerhalder S. Influence of Educational Interventions on the Test Ordering Patterns of Physicians . Ottawa, Ontario: Canadian Coordinating Office for Health Technology Assessment; 1992.
135.
Altshuler CH. Data utilization, not data acquisition, is the main problem.  Clin Chem.1994;40:1616-1620.
Toward Optimal Laboratory Use
August 12, 1998

Do We Know What Inappropriate Laboratory Utilization Is?A Systematic Review of Laboratory Clinical Audits

Author Affiliations

From the Institute for Clinical Evaluative Sciences, North York, Ontario (Drs van Walraven and Naylor), and the University of Toronto, Toronto, Ontario (Dr Naylor).

JAMA. 1998;280(6):550-558. doi:10.1001/jama.280.6.550
Objective.—

Objective.— Laboratory utilization has steadily increased, and some studies suggest inappropriate utilization. Therefore, we wished to assess studies that measure inappropriate laboratory use in light of methodological criteria.

Design.— Systematic review of published studies.

Data Sources.— MEDLINE, HEALTHSTAR, and EMBASE databases were searched from 1966 to September 1997 using a broad and inclusive strategy with no language restriction. In addition, the references of all retrieved studies and 3 textbooks on diagnostic testing were hand-searched.

Study Selection.— All studies that provided and applied criteria for inappropriate laboratory use.

Data Extraction.— Studies were categorized based on whether the criteria were implicit (objective criteria for inappropriate utilization not provided or very broad) or explicit. Guidelines for evaluation were applied to each study by a single reviewer.

Data Synthesis.— Forty-four eligible studies were identified. Eleven studies used implicit criteria for inappropriate laboratory utilization and contained small numbers of patients or physicians. Most did not adequately assess the reliability of the implicit criteria. Thirty-three studies used explicit criteria based on the appropriateness of test choice, frequency, and timing, as well as the probability of a positive result. There were large variations in the estimates of inappropriate laboratory use (4.5%-95%). Evidence supporting the explicit criteria was frequently weak by the standards suggested for therapeutic maneuvers, but was nonetheless compelling based on principles of physiology, pharmacology, and probability.

Conclusions.— Many studies identify inappropriate laboratory use based on implicit or explicit criteria that do not meet methodological standards suggested for audits of therapeutic maneuvers. Researchers should develop alternative evidentiary standards for measuring inappropriateness of laboratory test use.

THE APPROPRIATE utilization of laboratory tests is necessary for optimal patient care. The utilization of diagnostic laboratories has increased over the last several decades in many medical jurisdictions around the world.13 In Ontario, Canada's largest province, the total number of tests performed annually has increased by more than 130% between 1976 and 1992.4 The cost of increased laboratory utilization has been considerable, rising from an annual expenditure (in inflation-adjusted dollars) of Can $33.29 per person in 1979 to Can $103.36 in 1992.5 This increased laboratory use is appropriate if it allows accurate diagnoses to be made, ideal therapy to be identified and monitored, and accurate prognoses to be established.

Perhaps because of this increased use, laboratory utilization has been analyzed extensively with studies focused primarily on physician-ordering practices.611 Many studies have suggested that inappropriate test ordering is a primary reason for increased laboratory use.1220 If inappropriate laboratory utilization does exist, it must be corrected for several reasons. First, inappropriate testing not only causes unnecessary patient discomfort, it also increases the likelihood of generating false positive results, which cause unnecessary worry and further investigation.21 Second, laboratory tests are examples of "little-ticket" health care technologies,22 which, because they are commonly used, cost the health care system large amounts of money.2225 Finally, inappropriate laboratory utilization may be associated with other inefficiencies in health care delivery. Identifying inadequacies in the use of laboratory services may disclose problems in other areas of health care.

These considerations highlight the importance of determining when inappropriate laboratory utilization occurs and raise the question, "Do we know what inappropriate laboratory utilization is?" Evidence-based users' guides to assessing the medical literature have focused on laboratory test performance characteristics,26 but have not directly addressed test utilization. With this systematic review, we set out to identify published studies that measured inappropriate laboratory utilization. We used guidelines for appraising utilization reviews27 to determine the validity of each study's criteria for assessing appropriateness and the application of the criteria. However, since those methodological guidelines are directed primarily toward therapeutic utilization reviews, we determined the studies' compliance with the guidelines as well as the applicability of the guidelines to diagnostic utilization reviews. Finally, we have identified directions for future research in this area.

METHODS

The MEDLINE database was searched from 1966 to September 1997 by crossing several subject headings (explode laboratories, diagnostic services, diagnostic services–routine, explode diagnosis–laboratory, or explode quality assurance–health care) with several topic headings (attitude of health personnel,physicians' practice patterns, explode guidelines, or utilization review) or text words (unnecessary, duplication, efficiency, inappropriate, overutilization, underutilization, quality control, quality assurance, guidelines, utilization, or utilization review). Citations were limited to human subjects but no language restriction was used. A very inclusive search strategy was used because laboratory utilization is a broad topic applying to almost all areas of medicine. Each citation's title and abstract was reviewed and all studies potentially concerned with efficient laboratory use or methods to decrease laboratory use were retrieved.

Studies were included if they specified criteria for inappropriate laboratory utilization and actually used them in an audit. Studies were excluded if they assessed radiological or pathological tests, dealt exclusively with laboratory quality control issues (eg, culturing urine from leaky specimen containers), or assessed the appropriateness of the test's urgency (eg, was a "stat" order appropriate?). Studies that developed criteria for inappropriate laboratory utilization2845 were excluded if the criteria were not applied to a second group of patients. Finally, we excluded studies that determined the appropriateness of screening tests (eg, preoperative screening batteries, preoperative bleeding times, blood cross-matching). We reasoned that, instead of determining the appropriateness of a test for a particular patient in a particular situation, studies addressing the appropriateness of screening tests determine whether a particular test is appropriate for a group of patients. Determining the appropriateness of screening tests has been addressed in several publications.46,47

These inclusion and exclusion criteria were applied to the references of all articles as well as citations retrieved from EMBASE and HEALTHSTAR databases with a search strategy similar to that used in MEDLINE. Finally, hand-searching was used to screen all articles in the Toward Optimal Laboratory Use series from JAMA and all references in 3 commonly cited textbooks on laboratory test use.4850

Descriptive data that were abstracted from each study included country of publication, study setting, laboratory tests assessed, and the number of patients and physicians included in the study. All articles were categorized based on whether the criteria for inappropriate laboratory use were implicit (objective criteria for inappropriate use not provided or criteria very broad, leaving much latitude for interpretation by the reviewer) or explicit. Guidelines for the evaluation of utilization reviews27 were applied to each study by a single reviewer. These guidelines help determine whether the criteria are valid and whether they are applied appropriately. Validity of the criteria was assessed based on 3 components: the quality of the evidence on which the criteria are based; the methods used to tap expert opinion; and whether compliance with the criteria is associated with better patient outcomes. The strength of criteria application was based on 2 components: the reliability of criteria use and whether the impact of uncertainty in their application was measured. Estimates for the prevalence of inappropriate laboratory utilization were documented. When interventions aimed at altering laboratory use were studied, only preintervention measurements of inappropriate laboratory use were determined.

RESULTS

Figure 1 illustrates results of the search strategy. Eighty-three articles potentially providing criteria for laboratory inappropriateness were identified and retrieved. Seventy-one of these studies were excluded because they provided no appropriateness criteria (n=24); were review articles (n=19); or studied screening tests (n=10), laboratory quality control (n=6), the appropriateness of laboratory test urgency (n=3), or other topics (n=9). Of the 24 studies without a definition of inappropriate utilization, 4 derived criteria but did not apply them to a separate group of patients,19,44,51,52 whereas the other studies measured laboratory use without an appropriateness assessment.30,5371 Several of these studies referred to "inappropriate" and "needless" laboratory use in the study title or abstract without providing a definition of these terms in the manuscript.54,56,57 Twelve articles from the MEDLINE search were appropriate for study inclusion. Thirty-two other articles were identified by searching EMBASE and HEALTHSTAR databases, reviewing references of all retrieved articles, and hand-searching journal series and textbooks on laboratory testing. Thirty of these were in the MEDLINE database but were not captured by our initial search strategy or were overlooked during the initial citation screening. Four articles were excluded because they dealt exclusively with serum sampling time without assessing the appropriateness of ordering the test itself.7275

Forty-four articles meeting our inclusion criteria were identified. One study76 presented explicit criteria for both serum calcium and lactate dehydrogenase, which are discussed separately. Criteria for inappropriate use were classified as implicit for 11 studies and explicit for 33 studies. Studies were published between 1965 and 1995. Most studies (70%) originated from the United States, with the remainder from the United Kingdom,7780 the Netherlands,81,82 Australia,8385 Canada,8688 and 1 each from Egypt89 and Thailand.90

Studies Using Implicit Appropriateness Criteria

The 11 studies that used implicit criteria for appropriateness are listed in Table 1. All of these studies were conducted in teaching institutions and, with 2 exceptions,90,92 studied laboratory utilization for patients in hospitals. The number of patients assessed in the studies ranged from 25 to 555. Four studies cited the number of physicians ordering tests, which ranged from 393 to 18.90 Investigations that were assessed in the studies included basic biochemistry tests alone,94,95 with hematology tests, 78,93,96 or with hematology and microbiology tests.79,89,90,97 It was unclear from Dickinson's article92 what investigations were assessed.

Validity of the Criteria

Each study's criteria were based partly on whether the test helped diagnose or treat patients (Table 1). As is the norm with implicit review, the criteria were subjective and relied on interpretation by the reviewer. Several studies added qualifiers to the criteria, apparently to increase the reliability of the reviews94,96,97 ; however, the qualifiers themselves left latitude for interpretation. Since the criteria were broad, it is not surprising that none were based on specific published evidence. Two studies92,96 used a consensus conference to develop the broad criteria for implicit review, but an explicit group process was not reported. None of the studies correlated compliance with the criteria and favorable patient outcomes. However, this methodological standard had limited applicability because the studies were assessing the ordering of basic laboratory tests in isolation.

Criteria Application

Most studies used medical record review or a summary of the medical record review90 to apply the criteria. All audit criteria required the chart reviewer to identify each patient's symptoms or diagnoses to determine test appropriateness. Most studies78,79,89,9296 used a single reviewer for medical record abstraction. Two studies clarified uncertainties with the opinion of a second clinician96 or interviews with the requesting physician.95 In the study by Bold and Corrin,80 the senior surgical residents from each team determined appropriateness of all tests, presumably based on their memory of each case.

Only 2 studies directly compared the classification of laboratory appropriateness made by different reviewers.78,97 In 1 study,78 42.8% of 339 laboratory tests were classified as inappropriate by a panel of 3 physicians compared with only 26.5% by a single pathologist. Bloomgarden and Sidel97 showed that agreement between 2 physician groups was significantly beyond chance (κ=0.63, P<.001). Dickinson92 determined that intrarater agreement was 97% when the same reviewer reabstracted the charts after 3 months. One study allowed test appropriateness to be classified as uncertain78 but no sensitivity analysis (ie, the effect of reclassifying uncertain categorizations on the study's conclusion) was reported on these data.

The proportion of inappropriate tests varied among studies. Dixon and Laszlo94 and Sandler79 had the highest estimates, citing more than 90% of tests as inappropriate. These studies arguably also had the most stringent assessment criteria (Table 1). Data provided by Young93 did not allow for an estimate of test inappropriateness by individual test type. The rest of the studies concluded that between 10% and 50% of tests were inappropriate. None of the studies reported confidence intervals for the prevalence of inappropriate tests.

Studies Using Explicit Appropriateness Criteria

These studies are presented in Table 2, Table 3, and Table 4 and are categorized according to the laboratory test studied. In contrast with the implicit group, these studies concentrated on single laboratory tests. With several exceptions,8183,98104 most studies were conducted on patients hospitalized within teaching centers. The number of patients varied among studies (range, 15-2897) but they generally contained more than the implicit group. Finally, the sampling frames of these studies were sometimes very narrow and often included patients only if they had testing at a particular frequency,76,105 with a completed form,86,106 or for a particular indication.84

Validity of the Criteria

Although the explicit criteria for inappropriateness appear diverse (Table 2, Table 3, and Table 4), several themes are common to them. Criteria were based on testing frequency,76,82,88,99,105,107,108 timing of the test in relation to previous medications and tests,84,86,87,102,103,108113 test choice compared with possible alternatives,101,114 clinical indications for the test,81,85,100,104,106,115118 or probability that a test result was abnormal.83,98 Two studies119,120 defined a single blood culture as inappropriate because distinguishing between a true positive blood culture and a contaminant can be difficult when only 1 culture is done. Flynn et al121 defined a single gentamicin level as inappropriate since peak and trough levels are usually needed for appropriate dosage adjustment.

Most criteria were based on published literature or local consensus. Several studies81,87,98,101,110,112114,119 justified the criteria used by citing a varying number of published articles on the topic. Schoenenberger et al109 documented the methods used for conducting a systematic review to derive their criteria. Three studies based criteria on guidelines published by the American College of Physicians108,115 and other specialty societies.111 Local expert opinion was specified for criteria derivation in some studies76,106,107,109,116118 but the methods used to elicit these opinions were not delineated. Rhyne and Gehlbach100 based their criteria for thyroid testing on a survey involving physicians in training. Boon-Falleur et al81 used a probability analysis associating symptoms with thyroid dysfunction to create criteria. Morris et al99 based their criteria on detailed data from a cohort of patients investigated at their hospital. Finally, fundamentals of physiology98 and pharmacology84,86,88,103,109112,121 were used for criteria in the remaining studies.

Criteria Application

Since explicit criteria were used, these studies had more options for data collection necessary for criteria application. Many used data from laboratory databases82,83,98,99,101,103,105,107,108,119121 or requisition forms completed by the ordering physician.81,84,86,106,113,114 The remainder of the studies76,87,88,100,104,109,112,115118 required medical record review. The data source for 2 studies was uncertain.85,102 Several76,100,109,111 determined abstraction reliability with a second medical record reviewer but only Schoenenberger et al109 measured agreement beyond chance (κ=0.61, P <.001). Two studies clarified data from chart review by interviewing the physician who ordered the questionable tests.106,114 None of the studies associated compliance with the criteria with positive patient outcomes. Bowman et al83 and Morris et al,99 however, showed that tests that complied with the criteria were significantly more likely to have abnormal results.

Prevalence of Inappropriate Laboratory Utilization

Estimates for the proportion of inappropriate tests varied extensively (range, 4.5%-95%). Tests with the highest estimates for inappropriate utilization included prothrombin time,115 calcium,76 cerebrospinal fluid analysis for VDRL test,98 and antiepileptic drug monitoring.88,109 The study by Saxena et al108 probably overestimates inappropriate laboratory use because the criteria used for timing of repeat testing were stringent (ie, it assumes a meaningful difference between a serial creatine kinase level measured at 10 hours vs one measured at 12 hours). Estimates for inappropriate laboratory utilization ranged from 5% to 50% in the other studies. It is plausible that criteria relying on data from requisition forms would have higher inappropriateness rates because physician documentation on the requisition could be very incomplete. However, studies using this method81,84,106,114 did not find a higher prevalence of inappropriateness.

COMMENT

Appropriate laboratory utilization is a cornerstone of optimal medical practice. Patients benefit from accurate diagnoses, proper therapeutic monitoring, and precise prognostications, all of which result from the use of sensible diagnostic technologies. Inappropriate laboratory utilization can not only harm patients, it is also expensive. The authors of all studies cited in this review deserve commendation for trying to measure its prevalence. A broad range of criteria were used for different laboratory tests in distinct clinical situations. The validity of the criteria and the reliability of their application varied among the studies. The prevalence of inappropriate laboratory use also varied extensively in its magnitude.

Some studies used implicit as opposed to explicit criteria. Implicit criteria are versatile because a single definition can be applied to a broad range of tests. They allow a more complete review of laboratory utilization because the reviewer can consider multiple components of a patient's situation when determining the appropriateness of a laboratory test. However, there are many drawbacks to using implicit appropriateness criteria, including interreviewer variability in their interpretation and application. Use of implicit criteria usually demands detailed review of medical records. If physicians do not completely document their clinical reasoning or the patient's findings, laboratory tests may be judged inappropriate. The review could then become more an assessment of clinical documentation than of laboratory utilization. Also, medical record review is time-consuming and expensive. This will decrease the number of diagnostic episodes that can be assessed and therefore decrease the scope and precision of any assessment of laboratory utilization. With explicit criteria, it is often possible to collect the necessary data from databases or data requisitions, thus avoiding many of the problems associated with retrospective medical record review. Finally, explicit criteria make it possible to determine appropriateness criteria using prospectively collected data (eg, from a data requisition form). Prospectively collected data help avoid bias due to omission of physician documentation.

Two studies94,96 classified tests as inappropriate if therapy did not change as a result of test ordering. This criterion is problematic because the reviewer determining test appropriateness could not know whether or how therapy could have changed if the test result was different. Also, it is unreasonable to expect a laboratory test to single-handedly alter therapy because test results are only 1 of several parameters used in medical decision making. Finally, even tests that do not change therapy may provide useful information to physicians and patients.

Two studies79,80 based implicit reviews partly on whether the test result was abnormal. This criterion is questionable because normal test results are the most common outcome with laboratory testing. In addition, this criterion leads to an impossible double bind. To have completely appropriate laboratory utilization, one would need to know in advance that each test result would be abnormal. However, if the test result is known in advance, then ordering the test may be superfluous and inappropriate.

Many studies that used explicit criteria did not comply with guidelines for establishing the validity of utilization review criteria.27 Poor study design or poor reporting could explain noncompliance with these guidelines. However, it is also arguable that these guidelines are not as readily applicable to diagnostic as to therapeutic utilization reviews. Diagnostic technologies are often not tested with the same evaluative rigor as many therapeutic technologies. This is understandable in light of the lower cost of many diagnostic tests, their use as only 1 element in clinical decision making, and the attendant uncertainties in their link to patients' outcomes. As a result, there is a paucity of high-level evidence (such as randomized controlled trials) assessing diagnostic technology on which audit criteria can be based.26 This could explain why less than half of the studies cited evidence on which their audit criteria were based. Morris et al99 compensated for this by conducting a cohort study at their own institution on which their appropriateness criteria were based.

Compliance with utilization review guidelines is problematic for audits of diagnostic technologies for other reasons. For criteria development, expert opinion should be tapped using an explicit, systematic, and reliable process when published evidence is lacking or needs supplementation.27 This could be more difficult in the diagnostic arena because test appropriateness is highly dependent on the pretest probability of the condition being sought. The number of clinical situations in which pretest probabilities have been rigorously determined is relatively small. Also, the subjective determination of these probabilities122 and diagnostic or risk thresholds123,124 varies widely among physicians. These difficult issues may explain why only a few studies tapped expert opinion to derive criteria and why none used systematic methods in doing so.125

Finally, the validity of the criteria is enhanced when their compliance is associated with improved patient outcomes.27 For example, Kravitz et al126 showed that patients meeting panel criteria for revascularization who did not undergo such procedures fared significantly worse than those who did. In contrast, it is difficult for diagnostic technologies to conform with this criterion because tests are often used to determine whether the primary prognosticator (ie, the disease) is present. Test use by itself may therefore not be associated with patient outcomes. To compensate for this, several studies tried to validate their criteria by associating criteria compliance with surrogate outcomes such as costs or the prevalence of abnormal test results.83,99 Other investigators dealt with the difficulty of complying with these accepted methods of establishing valid audit criteria by basing them on physiological98 or pharmacological84,127 tenets. In our view, these are defensible alternatives, since pharmacokinetic and physiological principles can readily confirm that some tests will be redundant if repeated too frequently, whereas other tests cannot yield valid information unless performed at a prespecified interval in relation to some therapeutic intervention or physiological change in the patient's status.

For these reasons, it is understandable that audits of laboratory test utilization infrequently comply with methodological guidelines for validation of appropriateness criteria. What, then, of compliance with guidelines for application of the criteria? As discussed herein, using chart review in a "reliable and unbiased" fashion27 is difficult given the limitations of physician documentation. Several studies avoided medical record review by using clinical data documented on laboratory requisitions,81,84,106,114,127 although none of these studies validated these data. Despite the potential problems with medical record review, only 4 studies used 2 or more reviewers and compared their abstraction results. None of the studies determined the effect of uncertainty on study results with sensitivity analysis. These issues are less of a concern when objective criteria are used.

Are the results of these utilization reviews generalizable? Almost all the studies took place in teaching centers on hospitalized patients and analyzed the ordering practices of a small number of physicians-in-training. This represents a very select patient and physician group. Hospitalized patients are less healthy than the ambulatory population and laboratory-requesting patterns for them may not represent those of ambulatory patients. Physicians in training have less experience in diagnostic strategies and may order tests less efficiently than established practitioners. Since laboratory use is dependent on both patient and physician factors, it is necessary to research other clinical settings to determine the extent of inappropriate laboratory utilization. This could be done by using the criteria that are applicable with data potentially found within laboratory and administrative databases.82,84,98,101,103,105,119121 Certainly, allusions to extensive inappropriate laboratory use128,129 should not be made without appropriate qualifiers.

This review has several limitations that should be mentioned. First, the study deliberately excluded all radiology, pathology, and screening tests. Determining how well inappropriate utilization is measured in these areas would merit future reviews. Second, quality of study design and quality of reporting can be distinct. It is possible that the reliability of some studies was compromised when criteria for definitions or comparisons among raters were edited from the manuscripts. Finally, the diverse nature of this topic makes it possible that some eligible studies were not identified with our search strategy. The observation that less than half of the studies were identified from the initial MEDLINE search increases this concern.

The studies in this review suggest a paradigm or model for classifying criteria that determine the appropriateness of laboratory use (Figure 2). Appropriateness criteria could be based on a direct comparison among laboratory tests or a single test in isolation. The former could be based on studies using methods suggested by Reid et al26 and would compare tests based on their operating characteristics such as sensitivity, specificity, and likelihood ratios. The latter criterion, determining a test's appropriateness without another test for comparison, could be based on physiological, pharmacological, or probabilistic data. Physiological criteria would be based on our knowledge of disease processes, such as the criteria of Albright et al for VDRL testing in cerebrospinal fluid.98 Pharmacological criteria would be based on known pharmacodynamic tenets, such as those by Pearce and Day84 and Bussey and Hoffman.103 Clinical probability-based criteria would identify scenarios in which test results would almost never be useful. This would occur when a test's operating characteristics (eg, positive or negative likelihood ratios) are too weak to significantly alter posttest probabilities. Finally, a test may be useless if its result is highly predictable. Cohort studies could be used to identify these situations.29,30,32,34,3740,43,44,130 We hope that our model will serve as a framework for the development of appropriateness criteria for future studies and that these studies will concentrate first on establishing valid criteria for determining test appropriateness before it is measured.

Where should we go from here? Identifying inappropriate laboratory use is important and requires valid methods. Promising criteria for inappropriate laboratory use need to be validated and then applied to divergent populations of patients and physicians using methodologically sound studies.27 Further research is necessary to measure test operating characteristics and methods of determining pretest probabilities accurately.131,132 When inappropriate laboratory utilization is identified, methods to rectify the problem must be developed, tested, and implemented.133,134 Although appropriate laboratory testing does not necessarily translate into appropriate utilization of test results,135 it is an important step toward optimal care of the patient and better performance of the health care system.

References
1.
Showstack JA, Schroeder SA, Matsumato MF. Changes in use of medical technologies, 1972-1977: a study of 10 inpatient diagnoses.  N Engl J Med.1982;306:706-712.
2.
Fowkes FG. Containing the use of diagnostic tests.  Br Med J (Clin Res Ed).1985;290:488.
3.
Danzon PM, Manning WG, Marquis MS. Factors affecting laboratory test use and prices.  Health Care Financ Rev.1984:5;23-32.
4.
 Salient Features of the Laboratory Industry in Ontario, 1992-93 . Toronto: Ontario Ministry of Health; 1995.
5.
 Laboratory Services Review—Report to the Ministry of Health . Toronto: Ontario Ministry of Health; 1994.
6.
Epstein AM, McNeil BJ. Relationship of beliefs and behavior in test ordering.  Am J Med.1986;80:865-870.
7.
Williams SV, Eisenberg JM, Pascale LA, Kitz DS. Physicians' perceptions about unnecessary diagnostic testing.  Inquiry.1982;19:363-370.
8.
McGillivray DL, Roberts-Brauer R, Kramer MS. Diagnostic test ordering in the evaluation of febrile children: physician and environmental factors.  AJDC.1993;147:870-874.
9.
Griffith III CH, Wilson JF, Desai NS, Rich EC. Does pediatric housestaff experience influence tests ordered for infants in the neonatal intensive care unit?  Crit Care Med.1997;25:704-709.
10.
Bugter-Maessen AM, Winkens RA, Grol RP.  et al.  Factors predicting differences among general practitioners in test ordering behaviour and in the response to feedback on test requests.  Fam Pract.1996;13:254-258.
11.
Hartley RM, Epstein AM, Harris CM, McNeil BJ. Differences in ambulatory test ordering in England and America: role of doctors' beliefs and attitudes.  Am J Med.1987;82:513-517.
12.
Perraro F, Rossi P, Liva C.  et al.  Inappropriate emergency test ordering in a general hospital: preliminary reports.  Qual Assur Health Care.1992;4:77-81.
13.
Bareford D, Hayling A. Inappropriate use of laboratory services: long term combined approach to modify request patterns.  BMJ.1990;301:1305-1307.
14.
Beck JR. Does feedback reduce inappropriate test ordering?  Arch Pathol Lab Med.1993;117:33-34.
15.
Mozes B, Lubin D, Modan B, Ben-Bassat I, Gitel SN, Halkin H. Evaluation of an intervention aimed at reducing inappropriate use of preoperative blood coagulation tests.  Arch Intern Med.1989;149:1836-1838.
16.
Dans PE, Cafferty L, Otter SE, Johnson RJ. Inappropriate use of the cerebrospinal fluid Venereal Disease Research Laboratory (VDRL) test to exclude neurosyphilis.  Ann Intern Med.1986;104:86-89.
17.
Siegel DL, Edelstein PH, Nachamkin I. Inappropriate testing for diarrheal diseases in the hospital.  JAMA.1990;263:979-982.
18.
Hyams KC. Inappropriate urine cultures in hospitalized patients receiving antibiotic therapy.  Arch Intern Med.1987;147:48-49.
19.
Bush TM, Shlotzhauer TL, Grove W. Serum complements: inappropriate use in patients with suspected rheumatic disease.  Arch Intern Med.1993;153:2363-2366.
20.
Wu AH. Reducing the inappropriate utilization of clinical laboratory tests.  Conn Med.1997;61:15-21.
21.
Rang M. The Ulysses syndrome.  Can Med Assoc J.1972;106:122-123.
22.
Moloney TW, Rogers DE. Medical technology: a different view of the contentious debate over costs.  N Engl J Med.1979;301:1413-1419.
23.
Fineberg HV, Pearlman LA. Low-cost medical practices.  Annu Rev Public Health.1982;3:225-248.
24.
Thamer M, Fitzpatrick KJ, Perry S. Medical and surgical supply costs: case study of a costly low-cost technology.  Int J Technol Assess Health Care.1997;13:526-536.
25.
Lewis S, Foreman J. Low-cost diagnostic technologies and clinical outcomes: the impact of inappropriate utilization.  Int J Technol Assess Health Care.1997;13:501-511.
26.
Reid MC, Lachs MS, Feinstein AR. Use of methodological standards in diagnostic test research: getting better but still not good.  JAMA.1995;274:645-651.
27.
Naylor CD, Guyatt GH.for the Evidence-Based Medicine Working Group.  Users' guides to the medical literature, XI: how to use an article about a clinical utilization review.  JAMA.1996;275:1435-1439.
28.
Clough JD, Calabrese LH, Valenzuela R. The ANA profile: quality and cost-effective laboratory utilization.  Cleve Clin J Med.1989;56:245-248.
29.
Valenstein P, Pfaller M, Yungbluth M. The use and abuse of routine stool microbiology: a College of American Pathologists Q-probes study of 601 institutions.  Arch Pathol Lab Med.1996;120:206-211.
30.
Chitkara YK, McCasland KA, Kenefic L. Development and implementation of cost-effective guidelines in the laboratory investigation of diarrhea in a community hospital.  Arch Intern Med.1996;156:1445-1448.
31.
Murray PR, Washington JA. Microscopic and bacteriologic analysis of expectorated sputum.  Mayo Clin Proc.1975;50:339-344.
32.
Crowson TW, Rich EC, Woolfrey BF, Connelly DP. Overutilization of cultures of CSF for mycobacteria.  JAMA.1984;251:70-72.
33.
Caldwell G, Kellett HA, Gow SM.  et al.  A new strategy for thyroid function testing.  Lancet.1985;1:1117-1119.
34.
Brady MT, Pacini DL, Budde CT, Connell MJ. Diagnostic studies of nosocomial diarrhea in children: assessing their use and value.  Am J Infect Control.1989;17:77-82.
35.
John R, Henley R, Lloyd G, Elder GH. Evaluation of a new strategy for detection of thyroid dysfunction in the routine laboratory.  Clin Chem.1988;34:1110-1114.
36.
Ibrahim H, McKenna MJ, Feldkamp CS. A thyroid testing algorithm: results of a pilot study.  Henry Ford Hosp Med J.1991;39:30-34.
37.
Hayward RA, Shapiro MF, Oye RK. Laboratory testing on cerebrospinal fluid: a reappraisal.  Lancet.1987;1:1-4.
38.
Church DL, Cadrain G, Kabani A, Jadavji T, Trevenen C. Practice guidelines for ordering stool cultures in a pediatric population.  Am J Clin Pathol.1995;103:149-153.
39.
Barbut F, Leluan P, Antoniotti G, Collignon A, Sedallian A, Petit JC. Value of routine stool cultures in hospitalized patients with diarrhea.  Eur J Clin Microbiol Infect Dis.1995;14:346-349.
40.
Asnis DS, Bresciani A, Ryan M, McArdle P, Mollura JL, Ilardi CF. Cost-effective approach to evaluation of diarrheal illness in hospitals.  J Clin Microbiol.1993;31:1675.
41.
Perkins MD, Mirrett S, Reller LB. Rapid bacterial antigen detection is not clinically useful.  J Clin Microbiol.1995;33:1486-1491.
42.
Martin TG, Elenbaas RM, Pingleton SH. Use of peak expiratory flow rates to eliminate unnecessary arterial blood gases in acute asthma.  Ann Emerg Med.1982;11:70-73.
43.
Wenz B, Lampasso JA. Eliminating unnecessary urine microscopy: results and performance characteristics of an algorithm based on chemical reagent strip testing.  Am J Clin Pathol.1989;92:78-81.
44.
Valenstein PN, Koepke JA. Unnecessary microscopy in routine urinalysis.  Am J Clin Pathol.1984;82:444-448.
45.
Rodewald LE, Woodin KA, Szilagyi PG, Arvan DA, Raubertas RF, Powell KR. Relevance of common tests of cerebrospinal fluid in screening for bacterial meningitis.  J Pediatr.1991;119:363-369.
46.
Earle C, Hebert PC. A reader's guide to the evaluation of screening studies.  Postgrad Med J.1996;72:77-83.
47.
Sackett DL, Haynes RB, Guyatt GH.  et al.  Clinical Epidemiology: A Basic Science for Clinical Medicine . 2nd ed. Boston, Mass: Little Brown & Co Inc; 1991:153-172.
48.
Sox Jr HC. Common Diagnostic Tests: Use and Interpretation . Philadelphia, Pa: American College of Physicians; 1987.
49.
Speicher CE. The Right Test: A Physician's Guide to Laboratory Medicine . 2nd ed. Philadelphia, Pa: WB Saunders Co; 1993.
50.
Panzer RJ, Black ER, Griner PF. Diagnostic Strategies For Common Medical Problems . Philadelphia, Pa: American College of Physicians; 1991.
51.
Turnbull TL, Vanden Hoek TL, Howes DS, Eisner RF. Utility of laboratory studies in the emergency department patient with a new-onset seizure.  Ann Emerg Med.1990;19:373-377.
52.
Wenz B, Gennis P, Canova C, Burns ER. The clinical utility of the leukocyte differential in emergency medicine.  Am J Clin Pathol.1986;86:298-303.
53.
Grivell AR, Forgie HJ, Fraser CG, Berry MN. League tables of biochemical laboratory costs: an attempt to modify requesting patterns.  Med J Aust.1982;2:326-328.
54.
Durand-Zaleski I, Rymer JC, Roudot-Thoraval F, Revuz J, Rosa J. Reducing unnecessary laboratory use with new test request form: example of tumour markers.  Lancet.1993;342:150-153.
55.
Tierney WM, McDonald CJ, Martin DK, Rogers MP. Computerized display of past test results: effect on outpatient testing.  Ann Intern Med.1987;107:569-574.
56.
Roberts DE, Bell DD, Ostryzniuk T.  et al.  Eliminating needless testing in intensive care: an information-based team management approach.  Crit Care Med.1993;21:1452-1458.
57.
Ohkubo A. Reducing unnecessary laboratory use through education and regulation.  Rinsho Byori.1995;43:647-652.
58.
Wachtel TJ, O'Sullivan P. Practice guidelines to reduce testing in the hospital.  J Gen Intern Med.1990;5:335-341.
59.
Nightingale PG, Peters M, Mutimer D, Neuberger JM. Effects of a computerised protocol management system on ordering of clinical tests.  Qual Health Care.1994;3:23-28.
60.
Durand-Zaleski I, Rymer JC, Rosa J, Revuz J. Reducing unnecessary laboratory use with a new test request form: the example of tumor markers. In: Abstracts of the Annual Meeting, International Society of Technology Assessment in Health Care . Vol 9. New York, NY: International Society of Technology Assessment in Health Care; 1997:181.
61.
Tierney WM, McDonald CJ, Hui SL, Martin DK. Computer predictions of abnormal test results: effects on outpatient testing.  JAMA.1988;259:1194-1198.
62.
Tierney WM, Miller ME, McDonald CJ. The effect on test ordering of informing physicians of the charges for outpatient diagnostic tests.  N Engl J Med.1990;322:1499-1504.
63.
Tierney WM, Miller ME, Overhage JM, McDonald CJ. Physician inpatient order writing on microcomputer workstations: effects on resource utilization.  JAMA.1993;269:379-383.
64.
Liptzin BA, Williams JS. Laboratory utilization on a university surgical service.  Surgery.1972;71:247-253.
65.
Daniels M, Schroeder SA. Variation among physicians in use of laboratory tests, II: relation to clinical productivity and outcomes of care.  Med Care.1977;15:482-487.
66.
Ley C, Le C, Olshen EM, Reingold AL. The use of serologic tests for Lyme disease in a prepaid health plan in California.  JAMA.1994;271:460-463.
67.
Jackson LM, Shanahan F, Cryan B, Bredin CP, Cronin CC. Appropriateness of laboratory tests: requests for atypical pneumonia serology in a teaching hospital.  Ir J Med Sci.1996;165:93-94.
68.
McLeish WA. Third leukocyte differential counts: utilization study in a general hospital.  Can Med Assoc J.1985;132:616-617.
69.
Seah CS. Unnecessary investigation in medical practice.  Singapore Med J.1983;24:243-247.
70.
Joly JG. Budgetary impact of overutilization of multianalyzer tests.  Union Med Can.1981;110:989-994.
71.
Lepage R, Lessard F, Bannon P, Gauthier Y, Miron M, Joly JG. Over-utilization of tests: the responsibility of the laboratory.  Union Med Can.1984;113:526-532.
72.
Vlasses PH, DiPiro CA, Chalupa D, Hopkins LE. Appropriateness of sampling time for selected serum drug assays.  Hosp Pharm.1982;17:371-373.
73.
Barth JH, Page MD, Bailey IR. An audit of dynamic function tests: errors in the timing of blood specimens.  Clin Endocrinol (Oxf).1993;39:97-100.
74.
Pitterle ME, Sorkness CA, Wiederholt JB. Use of serum drug concentrations in outpatient clinics.  Am J Hosp Pharm.1985;42:1547-1552.
75.
Gotz VP, Lopez LM, Reeves SE, Morrow ME. Clinical utilization of serum theophylline concentrations in a university-affiliated hospital.  Drug Intell Clin Pharm.1982;16:872-874.
76.
Eisenberg JM, Williams SV, Garner L, Viale R, Smits H. Computer-based audit to detect and correct overutilization of laboratory tests.  Med Care.1977;15:915-921.
77.
Gama R, Nightingale PG, Ratcliffe JG. Effect of educational feedback on clinicians' requesting of cardiac enzymes.  Ann Clin Biochem.1992;29(pt 2):224-225.
78.
McConnell TS, Berger PR, Dayton HH, Umland BE, Skipper BE. Professional review of laboratory utilization.  Hum Pathol.1982;13:399-403.
79.
Sandler G. Do emergency tests help in the management of acute medical admissions?  Br Med J (Clin Res Ed).1984;289:973-977.
80.
Bold AM, Corrin B. Use and abuse of clinical chemistry in surgery.  Br Med J.1965;5469:1051-1052.
81.
Boon-Falleur L, Sokal E, Nightingale PG.  et al.  Utilization of laboratory resources: developments in knowledge-based ordering systems.  Int J Biomed Comput.1995;40:17-30.
82.
Branger PJ, Van Oers RJ, van der Wouden JC, van der Lei J. Laboratory services utilization: a survey of repeat investigations in ambulatory care.  Neth J Med.1995;47:208-213.
83.
Bowman RA, Bowman JM, Arrow SA, Riley TV. Selective criteria for the microbiological examination of faecal specimens.  J Clin Pathol.1992;45:838-839.
84.
Pearce GA, Day RO. Compliance with criteria necessary for effective drug concentration monitoring.  Ther Drug Monit.1990;12:250-257.
85.
Penglis S, Martin ED. Evaluation of serum drug level monitoring on a general medical ward.  Aust J Hosp Pharm.1984;14:138-141.
86.
Clague HW, Twum-Barima Y, Carruthers SG. An audit of requests for therapeutic drug monitoring of digoxin: problems and pitfalls.  Ther Drug Monit.1983;5:249-254.
87.
Wing DS, Duff HJ. The impact of a therapeutic drug monitoring program for phenytoin.  Ther Drug Monit.1989;11:32-37.
88.
Levine M, McCollom R, Chang T, Orr J. Evaluation of serum phenytoin monitoring in an acute care setting.  Ther Drug Monit.1988;10:50-57.
89.
Rafeh N, el-Tobgi D. Clinical practice and the use of laboratory tests at the May 15 Hospital in Egypt.  Int J Qual Health Care.1995;7:25-30.
90.
Ruangkanchanasetr S. Laboratory investigation utilization in pediatric out-patient department, Ramathibodi Hospital.  J Med Assoc Thai.1993;76(suppl 2):194-208.
91.
Myers LP, Chapman SA, McPhee SJ, Schroeder SA, Leong JK, Martin AR. Which hospital services are most overused? results from a medical audit.  West J Med.1985;143:397-398.
92.
Dickinson JC. From process to policy: a generic prescription for test over-utilization in the emergency department.  Fam Pract Res J.1987;7:12-21.
93.
Young DW. An aid to reducing unnecessary investigations.  BMJ.1980;281:1610-1611.
94.
Dixon RH, Laszlo J. Utilization of clinical chemistry services by medical house staff: an analysis.  Arch Intern Med.1974;134:1064-1067.
95.
Baigelman W, Bellin SJ, Cupples LA, Dombrowski D, Coldiron J. Overutilization of serum electrolyte determinations in critical care units: savings may be more apparent than real but what is real is of increasing importance.  Intensive Care Med.1985;11:304-308.
96.
Gortmaker SL, Bickford AF, Mathewson HO, Dumbaugh K, Tirrell PC. A successful experiment to reduce unnecessary laboratory use in a community hospital.  Med Care.1988;26:631-642.
97.
Bloomgarden Z, Sidel VW. Evaluation of utilization of laboratory tests in a hospital emergency room.  Am J Public Health.1980;70:525-528.
98.
Albright Jr RE, Christenson RH, Emlet JL.  et al.  Issues in cerebrospinal fluid management: CSF Venereal Disease Research Laboratory testing.  Am J Clin Pathol.1991;95:397-401.
99.
Morris AJ, Wilson ML, Reller LB. Application of rejection criteria for stool ovum and parasite examinations.  J Clin Microbiol.1992;30:3213-3216.
100.
Rhyne RL, Gehlbach SH. Effects of an educational feedback strategy on physician utilization of thyroid function panels.  J Fam Pract.1979;8:1003-1007.
101.
Schectman JM, Elinsky EG, Pawlson LG. Effect of education and feedback on thyroid function testing strategies of primary care clinicians.  Arch Intern Med.1991;151:2163-2166.
102.
Greenlaw CW, Geiger GS, Haug III MT. Use of digoxin serum assays in a nonteaching hospital.  Am J Hosp Pharm.1980;37:1466-1470.
103.
Bussey HI, Hoffman EW. A prospective evaluation of therapeutic drug monitoring.  Ther Drug Monit.1983;5:245-248.
104.
Greenlaw CW, Blough SS, Haugen RK. Aminoglycoside serum assays restricted through a pharmacy program.  Am J Hosp Pharm.1979;36:1080-1083.
105.
Valenstein P, Leiken A, Lehmann C. Test-ordering by multiple physicians increases unnecessary laboratory examinations.  Arch Pathol Lab Med.1988;112:238-241.
106.
Bartlett RC, Mazens-Sullivan M, Tetreault JZ, Lobel S, Nivard J. Evolving approaches to management of quality in clinical microbiology.  Clin Microbiol Rev.1994;7:55-88.
107.
Studnicki J, Bradham DD, Marshburn J, Foulis PR, Straumfjord JV. A feedback system for reducing excessive laboratory tests.  Arch Pathol Lab Med.1993;117:35-39.
108.
Saxena S, Anderson DW, Kaufman RL, Hannah JA, Wong ET. Quality assurance study of cardiac isoenzyme utilization in a large teaching hospital.  Arch Pathol Lab Med.1993;117:180-183.
109.
Schoenenberger RA, Tanasijevic MJ, Jha A, Bates DW. Appropriateness of antiepileptic drug level monitoring.  JAMA.1995;274:1622-1626.
110.
Slaughter RL, Schneider PJ, Visconti JA. Appropriateness of the use of serum digoxin and digitoxin assays.  Am J Hosp Pharm.1978;35:1376-1379.
111.
Guernsey BG, Ingrim NB, Hokanson JA.  et al.  A utilization review of theophylline assays: sampling patterns and use.  Drug Intell Clin Pharm.1984;18:906-912.
112.
Sargenti C, Zelman L, Beauclair T, Garrard E, Boehm R. Evaluation of appropriateness and interpretation of serum theophylline assays.  Drug Intell Clin Pharm.1985;19:380-384.
113.
Ives TJ, Parry JL, Gwyther RE. Serum drug level utilization review in a family medicine residency program.  J Fam Pract.1984;19:507-512.
114.
Finn Jr AF, Valenstein PN, Burke MD. Alteration of physicians' orders by nonphysicians.  JAMA.1988;259:2549-2552.
115.
Erban SB, Kinman JL, Schwartz JS. Routine use of the prothrombin and partial thromboplastin times.  JAMA.1989;262:2428-2432.
116.
Kroenke K, Hanley JF, Copley JB.  et al.  Improving house staff ordering of three common laboratory tests: reductions in test ordering need not result in underutilization.  Med Care.1987;25:928-935.
117.
Goldberg GA, Abbott JA. Explicit criteria for use of laboratory tests.  Ann Intern Med.1974;81:857-858.
118.
Levin B, Cohen SS, Birmingham PH. Effect of pharmacist intervention on the use of serum drug assays.  Am J Hosp Pharm.1981;38:845-851.
119.
Schifman RB, Strand CL, Braun E, Louis-Charles A, Spark RP, Fried ML. Solitary blood cultures as a quality assurance indicator.  Qual Assur Utilization Rev.1991;6:132-137.
120.
Gross PA, Van Antwerpen CL, Hess WA, Reilly KA. Use and abuse of blood cultures: program to limit use.  Am J Infect Control.1988;16:114-117.
121.
Flynn TW, Pevonka MP, Yost RL, Weber CE, Stewart RB. Use of serum gentamicin levels in hospitalized patients.  Am J Hosp Pharm.1978;35:806-808.
122.
Dolan JG, Bordley DR, Mushlin AI. An evaluation of clinicians' subjective prior probability estimates.  Med Decis Making.1986;6:216-223.
123.
Nightingale SD. Risk preference and admitting rates of emergency room physicians.  Med Care.1988;26:84-87.
124.
Nightingale SD, Grant M. Risk preference and decision making in critical care situations.  Chest.1988;93:684-687.
125.
Brook RH, Chassin MR, Fink A, Solomon DH, Kosecoff J, Park RE. A method for the detailed assessment of the appropriateness of medical technologies.  Int J Technol Assess Health Care.1986;2:53-63.
126.
Kravitz RL, Laouri M, Kahan JP.  et al.  Validity of criteria used for detecting underuse of coronary revascularization.  JAMA.1995;274:632-638.
127.
Gibb I, Cowan JC, Parnham AJ, Thomas TH. Use and misuse of a digoxin assay service.  BMJ.1986;293:678-680.
128.
Lyon AW, Greenway DC, Hindmarsh JT. A strategy to promote rational clinical chemistry test utilization.  Am J Clin Pathol.1995;103:718-724.
129.
Leese B. Is There Too Much Laboratory Testing?  York, England: University of York; 1991:79.
130.
Fan K, Morris AJ, Reller LB. Application of rejection criteria for stool cultures for bacterial enteric pathogens.  J Clin Microbiol.1993;31:2233-2235.
131.
Wells PS, Hirsh J, Anderson DR.  et al.  Accuracy of clinical assessment of deep-vein thrombosis.  Lancet.1995;345:1326-1330.
132.
Stiell IG, Wells G, Laupacis A.  et al.  Multicentre trial to introduce the Ottawa ankle rules for use of radiography in acute ankle injuries.  BMJ.1995;311:594-597.
133.
Axt-Adam P, van der Wouden JC, van der Does E. Influencing behavior of physicians ordering laboratory tests: a literature study.  Med Care.1993;31:784-794.
134.
Menon D, Hommerhalder S. Influence of Educational Interventions on the Test Ordering Patterns of Physicians . Ottawa, Ontario: Canadian Coordinating Office for Health Technology Assessment; 1992.
135.
Altshuler CH. Data utilization, not data acquisition, is the main problem.  Clin Chem.1994;40:1616-1620.
×