[Skip to Content]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address 54.205.87.3. Please contact the publisher to request reinstatement.
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
Download PDF
Image description not available.
Figure 1.—Utilization rates for control laboratory tests. Rates for copper and aldolase tests are too small to be seen on the graph's scale.
Image description not available.
Figure 2.—Effects of interventions on utilization of the erythrocyte sedimentation rate test (ESR) and urinalysis with and without microscopy. Erythrocyte sedimentation rate test was removed from the laboratory requisition form in July 1994. A guideline recommending that ESR not be used to screen for illness was introduced in August 1994 and November 1995. In July 1994, policy was changed to exclude urine microscopy from routine urinalysis. Guidelines stressing the disutility of urine microscopy in asymptomatic patients were circulated in August 1994 and November 1995. Predicted rates of urinalysis without microscopy are not presented because of the graph's scale.
Image description not available.
Figure 3.—Effects of interventions on utilization of urea and creatinine tests. In November 1995, a guideline was circulated that recommended a creatinine test for measuring renal function and stressed that, in most cases, a urea test was unnecessary. In January 1996, the urea test was removed from the requisition form.
Image description not available.
Figure 4.—Effects of interventions on utilization of iron and ferritin tests. In April 1995, laboratories stopped performing serum iron tests when ordered with ferritin tests. Guidelines were produced at the same time encouraging the use of ferritin tests for investigating iron deficiency.
Image description not available.
Figure 5.—Effects of interventions on utilization of total thyroxine and thyroid-stimulating hormone (TSH) tests. In January and March 1992, guidelines were disseminated encouraging the use of the TSH test instead of the total thyroxine test. In November 1993, funding for total thyroxine tests was stopped. In August 1996, the TSH test was removed from the requisition form and a brief guideline was circulated discouraging its use as a screening test.
Summary of Interventions*
Image description not available.
1.
Leese B. Is There Too Much Laboratory Testing? York, England: University of York; 1991:79.
2.
van Walraven C, Naylor CD. Do we know what inappropriate laboratory utilization is?  JAMA.1998;280:550-558.
3.
Axt-Adam P, van der Wouden JC, van der Does E. Influencing behavior of physicians ordering laboratory tests.  Med Care.1993;31:784-794.
4.
Menon D, Hommerhalder S. Influence of Educational Interventions on the Test Ordering Patterns of Physicians. Ottawa, Ontario: Canadian Coordinating Office for Health Technology Assessment; 1992.
5.
Davidoff F, Goodspeed R, Clive J. Changing test ordering behavior.  Med Care.1989;27:45-58.
6.
Thompson RS, Kirz HL, Gold RA. Changes in physician behavior and cost savings associated with organizational recommendations on the use of "routine" chest X rays and multichannel blood tests.  Prev Med.1983;12:385-396.
7.
Mozes B, Lubin D, Modan B, Ben-Bassat I, Gitel SN, Halkin H. Evaluation of an intervention aimed at reducing inappropriate use of preoperative blood coagulation tests.  Arch Intern Med.1989;149:1836-1838.
8.
Wong ET, McCarron MM, Shaw Jr ST. Ordering of laboratory tests in a teaching hospital: can it be improved?  JAMA.1983;249:3076-3080.
9.
Zaat JO, van Eijk JT, Bonte HA. Laboratory test form design influences test ordering by general practitioners in the Netherlands.  Med Care.1992;30:189-198.
10.
Durand-Zaleski I, Rymer JC, Roudot-Thoraval F, Revuz J, Rosa J. Reducing unnecessary laboratory use with new test request form: example of tumour markers.  Lancet.1993;342:150-153.
11.
Novich M, Gillis L, Tauber AI. The laboratory test justified.  Am J Clin Pathol.1985;84:756-759.
12.
Dixon RH, Laszlo J. Utilization of clinical chemistry services by medical house staff: an analysis.  Arch Intern Med.1974;134:1064-1067.
13.
Ohkubo A. Reducing unnecessary laboratory use through education and regulation.  Rinsho Byori.1995;43:647-652.
14.
 Salient Features of the Laboratory Industry in Ontario, 1992-93.  Toronto: Ontario Ministry of Health; 1995.
15.
MacDonald D, From G, Luxton AGW, Naylor CD.for the Working Party on Testing Strategies in Thyroid Disease.  Report of the Working Party on Testing Strategies in Thyroid Disease. Toronto: Ontario Ministry of Health; 1992.
16.
Davis DA, Taylor-Vaisey A. Translating guidelines into practice.  CMAJ.1997;157:408-416.
17.
Oxman AD, Thomson MA, Davis DA, Haynes RB. No magic bullets.  CMAJ.1995;153:1423-1431.
18.
Avorn J, Soumerai SB. Improving drug-therapy decisions through educational outreach.  N Engl J Med.1983;308:1457-1463.
19.
Cohen SJ, Christen AG, Katz BP.  et al.  Counseling medical and dental patients about cigarette smoking.  Am J Public Health.1987;77:313-316.
20.
Everett GD, deBlois CS, Chang PF, Holets T. Effect of cost education, cost audits, and faculty chart review on the use of laboratory services.  Arch Intern Med.1983;143:942-944.
21.
Putnam RW, Curry L. Physicians' participation in establishing criteria for hypertension management in the office.  CMAJ.1989;140:806-809.
22.
Sibley JC, Sackett DL, Neufeld V, Gerrard B, Rudnick KV, Fraser W. A randomized trial of continuing medical education.  N Engl J Med.1982;306:511-515.
23.
Lundberg GD. Laboratory request forms (menus) that guide and teach.  JAMA.1983;249:3075.
24.
Moloney TW, Rogers DE. Medical technology—a different view of the contentious debate over costs.  N Engl J Med.1979;301:1413-1419.
Toward Optimal Laboratory Use
December 16, 1998

Effect of Population-Based Interventions on Laboratory UtilizationA Time-Series Analysis

Author Affiliations

From the Clinical Epidemiology Unit, Ottawa Hospital, Ottawa, Ontario (Dr van Walraven); the University of Toronto (Dr Goel) and Price-Waterhouse (Dr Chan), Toronto, Ontario; and the Institute for Clinical Evaluative Sciences, North York, Ontario (Drs van Walraven, Goel, and Chan). Dr van Walraven was an R. Samuel McLaughlin Foundation research fellow at the Institute for Clinical Evaluative Sciences when this study was conducted.

 

Toward Optimal Laboratory Use Section Editor: George D. Lundberg, MD, Editor, JAMA.

JAMA. 1998;280(23):2028-2033. doi:10.1001/jama.280.23.2028
Context.—

Context.— Previous studies have identified methods of decreasing laboratory utilization. However, most were hospital-based, relatively small, single-centered, or of limited duration.

Objective.— To determine the effect of 3 population-based interventions (physician guidelines, laboratory requisition form modification, and changes to funding policy) on laboratory utilization in Ontario.

Design.— Interventional time-series analysis in which data analysis was based on all claims made to the Ontario Health Insurance Program between July 1, 1991, and April 1997 for laboratory tests affected by the interventions.

Setting.— All clinical laboratories (not based in hospitals) in Ontario.

Interventions.— Physician guidelines, modification of laboratory requisition form, and changes in funding policy for the use of the erythrocyte sedimentation rate test (ESR), microscopic urinalysis, tests for renal function, iron stores, serum urea, and serum iron determinations, and tests for thyroid dysfunction (total thyroxine and thyroid-stimulating hormone [TSH]).

Main Outcome Measures.— Change from 1991 to 1997 in utilization rates of ESR, microscopic urinalysis, serum urea and iron determinations, and tests for total thyroxine and TSH.

Results.— Age- and sex-standardized rates for laboratory tests unaffected by the interventions were stable during the study period. Utilization of ESR and urea determination decreased by 58% (P<.001) and 57% (P<.001), respectively, after they were removed from the requisition form and guidelines discouraging their use were disseminated. Rates for urinalyses without microscopy increased by 1700% (P<.001), while microscopic urinalysis decreased by 14% (P<.001), after a policy change eliminated microscopic urinalysis from routine urinalysis. Rates of iron determination declined by 80% (P<.001) and ferritin rates increased by 34% (P = .05) when policy changes eliminated iron testing when ordered with ferritin and guidelines advocating ferritin alone for investigating iron deficiency were disseminated. Utilization of total thyroxine testing declined by 96% (P = .02) when the provincial health plan stopped its funding. When TSH was removed from the laboratory requisition form, a 12% decline (P = .03) in its use was observed. Through April 1997, these interventions saved more than 625,000 tests or $210,400.

Conclusions.— The combination of guideline dissemination, laboratory requisition form modification, and changes to funding policy was associated with significant reductions in laboratory utilization.

THE UTILIZATION OF laboratory services has increased during the last several decades in many health care jurisdictions. It is often assumed that some of this use is inappropriate,1 although the evidence supporting this supposition is weak.2 Interventions to improve laboratory utilization include physician education, laboratory requisition form changes, and policies concerning laboratory test ordering. Studies have concluded that educational interventions have mixed effects on laboratory test use.38 Significant decreases in test rates were seen when laboratory requisition forms were modified to contain fewer test choices,9 presented tests in physiologically sensible groups,8,10 or required ordering physicians to justify the need for the test.11 Some studies have shown that policies that prohibit particular tests in particular situations7 or limit the allowable total number of investigations12,13 are effective in decreasing use.

As a group, however, these studies are limited by small numbers of patients and physicians. Also, most were isolated to single teaching centers, studied complex interventions, or had follow-up periods that were too short to determine whether the intervention was persistently effective. Thus, the validity and generalizability of these findings are uncertain.

Between 1976 and 1993 in Ontario, laboratory testing increased from 9.4 to 17.4 tests per person. To address this trend, the Ontario Ministry of Health and the Ontario Association of Medical Laboratories identified areas within the laboratory sector where utilization could be suboptimal. Expert panels were convened to identify optimal laboratory utilization in these areas. Guidelines were generated and disseminated to practicing physicians and were often associated with other interventions to alter laboratory use, such as requisition form modification and funding-policy changes. The purpose of this study was to assess the impact of these population-based interventions on laboratory utilization in Ontario.

METHODS

We identified all guidelines that addressed physician ordering of laboratory tests produced by the Ontario Association of Medical Laboratories (Table 1). The guidelines were developed between 1991 and 1997 by independent expert panels and were distributed to all physicians who ordered tests from any private laboratory in Ontario. All changes to the laboratory requisition form and laboratory test funding were identified from bulletins produced by the Ministry of Health. All private laboratory services in Ontario must be requested using the Ministry of Health laboratory requisition form. When the form was modified, physicians were instructed to exhaust supplies of previous versions of the form before using the modified requisitions. The interventions addressed the utilization of the erythrocyte sedimentation rate test (ESR), urinalysis, and testing for renal dysfunction, iron deficiency, and thyroid dysfunction (Table 1).

All permanent residents in Ontario are covered by the Ontario Health Insurance Program (OHIP). Laboratory tests are recorded in the OHIP database, with each claim citing the patient, test provided, and service date. We retrieved all claims made to OHIP between July 1, 1991, and April 1997 for laboratory tests potentially affected by the interventions, including ESR; urinalysis with and without microscopy; and tests of serum urea and creatinine, serum ferritin and iron, total thyroxine, triiodothyronine resin uptake, and thyroid-stimulating hormone (TSH). As controls, 6 laboratory tests not affected by the interventions were also examined. These tests were chosen because they represented high (serum glucose and hemoglobin), middle (serum sodium and uric acid), and low (serum copper and aldolase) utilization rates.

Services for patients without valid OHIP numbers and services provided out of province were excluded. Almost all laboratory services performed by hospital-based laboratories were covered by each hospital's global budget and were therefore not captured by the OHIP database. In 1992, approximately 45% of all laboratory services in Ontario were provided outside of hospitals by commercial laboratories.14 However, the majority of hospital-based laboratories provide service to inpatients. Physicians in private practice largely use commercial laboratories.

Patient health care numbers were used to identify each patient's age and sex from the Registered Persons Database. Claims were grouped by sex, 5-year age intervals, and 28-day time intervals beginning on July 1, 1991. Therefore, each 4-week time interval had an equal number of weekends and approximately equal number of working days, with each calendar year having 13 intervals. Statistics Canada census data provided the number of people within each sex and age category in Ontario on July 1 of each study year. To determine the population for each time interval's midpoint for each age and sex stratum, linear extrapolation between July 1 estimates for successive years was used. Crude and age- and sex-standardized rates for each laboratory test were calculated, but only the latter are reported.

Time-series analysis was used to determine if each intervention was associated with significant changes in laboratory utilization rates. Observations having a temporal sequence are often autocorrelated (ie, the value at time x is affected by the value at time x − 1). As a result, the error terms of these observations are not independent, making simple regression inappropriate for their analysis. Time-series analysis is a collection of techniques for modeling autocorrelation in temporally sequenced data, thereby permitting traditional regression and inference testing.

Autoregressive integrated moving average (ARIMA) modeling was used in this study. Each intervention was represented by a dummy series, with 0 and 1 representing preintervention and postintervention, respectively. When interventions were introduced within 1 month of each other, they were analyzed together. If interventions were introduced within 2 months of each other but were highly correlated with each other in the final ARIMA model (correlation coefficient for the 2 interventions, >0.6), they were also analyzed together. Test and intervention series were cross-correlated and any seasonal variation or linear trends were removed by differencing rates between successive years or intervals, respectively. Autoregressive and moving average parameters for each ARIMA model of the deseasoned and detrended time series were identified by examining the partial autocorrelation function and the autocorrelation function, respectively. The model that provided the highest lag-6 and lag-12 q statistic was chosen. If the t value of the intervention parameter in the final ARIMA model exceeded a P value of .05, the intervention was considered to be significantly associated with changes in laboratory utilization. All models used to determine the significance of the interventions in this study fit the data well, with median lag-6 and lag-12 q statistics of 0.83 (range, 0.46-0.92) and 0.82 (range, 0.44-0.97), respectively.

To measure laboratory utilization changes associated with significant interventions, ARIMA modeling was used to forecast preintervention utilization rates with 95% confidence intervals (CIs). Differences between observed and predicted rates were determined and the total number of tests avoided (or added) by the interventions were multiplied by their cost, determined by the provincial laboratory fee schedule. All costs were expressed in US dollars (Can $1 ≈ US $0.65). All analyses were performed using SAS for Windows, Version 6.11 (SAS Institute Inc, Cary, NC).

RESULTS

Age- and sex-standardized utilization rates for the control tests from July 1991 through April 1997 are presented in Figure 1. Consistent seasonal variation is seen in each series, with nadirs coinciding with late summer and the Christmas holiday. The rates exhibit no convincing trend throughout the period. Rates for copper and aldolase utilization had similar variations but are not seen in Figure 1 because of the graph's scale.

Utilization of ESR is shown in Figure 2. In July 1994, the ESR "tick-box" was removed from the laboratory requisition form, followed in August 1994 by a guideline discouraging ESR screening for asymptomatic patients. Utilization dropped significantly by 58% (P<.001), with rates decreasing from approximately 2000 to 500 per 100,000 persons. The guideline was reissued in November 1995 with no effect (P = .97). Between June 1994 and April 1997, the guideline and requisition form changes were associated with avoiding 36,697 tests (95% CI, 26,312-47,083).

Figure 2 also displays the effects of interventions on urine microscopy. Prior to July 1994, microscopy was routine with each urinalysis. Policy was then changed, making microscopy at routine urinalysis unnecessary unless specifically ordered by the physician. In August 1994, a guideline stressing the futility of urine microscopy in asymptomatic patients was distributed. These interventions resulted in an abrupt increase (approximately 1400%) in urinalysis without microscopy (P<.001) along with a decrease in urinalysis with microscopy of 14% (P<.001). The increased number of urinalyses without microscopy was significant (12,216 [95% CI, 12,028-12,403]), but the estimated number of avoided urine microscopies was not significant (9690 [95% CI, −4570 to 23,951]). When the guidelines were reissued in November 1995, rates for urinalyses both with and without microscopy decreased by 5% and 9%, respectively, but this did not reach statistical significance (P = .06 and .09, respectively).

In November 1995, a guideline was disseminated recommending creatinine testing without, in most situations, serum urea testing for assessing renal function. In January 1996, urea testing was removed from the laboratory requisition form. Significant decreases in urea test rates were associated with both interventions (P<.001 for both; Figure 3), and overall urea test utilization decreased by 57%, with 17,347 tests (95% CI, 13,325-21,369) avoided. In November, there was a significant but less extensive increase in the use of creatinine testing (P = .01; Figure 3). The increased number of creatinine tests during the next 16 intervals did not significantly exceed forecasted rates (2571 tests [95% CI, −1498 to 6640]). We were unable to determine if removal of urea testing from the requisition form in January 1996 affected creatinine test utilization because it was highly correlated (>0.6) with the November 1995 guidelines.

Rates for iron store investigations rates are shown in Figure 4. In April 1995, laboratories stopped performing serum iron or total iron binding capacity tests when ordered with serum ferritin tests (unless the physician contacted the laboratory to explain why both tests were necessary). At the same time, a guideline recommending ferritin testing alone to investigate iron deficiency was disseminated. Slightly later (June 1995), a second guideline essentially repeating this message was released. In the final ARIMA model, these interventions were highly correlated (0.7) and are therefore analyzed together. The interventions were significantly associated with an 80% decrease in serum iron test rates (P<.001) and an 80% increase in ferritin test rates (P = .05). Overall, the intervention was associated with decreased test utilization because the increased number of ferritin tests (725 [95% CI, −712 to 2163]) was overshadowed by a significant decrease in the number of serum iron tests (3847 [95% CI, 2600-5095]).

Figure 5 shows utilization rates for thyroid testing. In January 1992, the Report of the Working Party on Testing Strategies in Thyroid Disease15 was released. A synopsis was disseminated to physicians 2 months later. This guideline recommended a sensitive TSH assay in lieu of total thyroxine and triiodothyronine resin uptake tests. Total thyroxine test utilization started decreasing at this time, but the significance of these guidelines could not be determined because too few observations were available prior to the report. As expected, total thyroxine test utilization decreased significantly by 96% when policy changes stopped funding for these tests in November 1993 (P = .02). These changes were associated with avoiding 4359 (95% CI, −14 to 23,430) and 3073 (−28 to 18,153) total thyroxine and triiodothyronine resin uptake tests, respectively.

The 1992 guidelines were also associated with increased use of TSH testing (Figure 5), but the 1993 policy changes did not significantly (P = .74) increase TSH test rates. Because of rising TSH test rates, the TSH "tick box" was removed from the requisition form in early August 1996, accompanied by a physician guideline discouraging the use of TSH testing to screen asymptomatic patients. This modification significantly decreased TSH test utilization by 12% (P = .03) and saved 2200 tests (95% CI, −1638 to 6039).

For interventions that significantly changed laboratory utilization rates, we calculated the difference between the number of tests ordered and the number forecasted by the preintervention time-series model through April 1997. For each test, this total difference was multiplied by the test cost, cited in the provincial fee schedule. When all interventions were considered together, we estimated that the interventions were associated with a decrease of 626,098 tests. This saved the system $210,400, including $29,664 in the final year of the study, ending in April 1997. This figure does not consider costs of physician visits and further investigations resulting from false-positive results avoided by the interventions. Because the unit cost per test in this study was low (range, $0.98-$13), a large number of tests were avoided to account for the final amount of money saved.

COMMENT

To our knowledge, this is the first population-level study measuring the effects of interventions aimed at modifying laboratory utilization. Over a 6-year period, we observed significant changes in the use of common laboratory tests. Combinations of practice guidelines, modifications to the laboratory requisition form, and funding policy changes were associated with significant decreases in the utilization of several tests. The effects of these interventions appeared to persist, avoided a large number of tests, and decreased costs.

One of the primary objectives of our study was to determine whether guideline dissemination changed laboratory utilization. Because most guidelines in this study were introduced concurrently with other interventions, their effects could not be determined independently. However, we observed 3 instances in which guideline dissemination was associated with significant changes in laboratory utilization, independent of cointerventions. In November 1995, a guideline recommending creatinine testing instead of urea testing for assessment of renal function was disseminated. This was associated with a significant decrease in urea testing rates (Figure 3). At the same time, a guideline was reissued dissuading the use of urinalysis for asymptomatic patients and was associated with a decrease in urinalysis rates. Finally, in April 1995, a guideline recommending ferritin testing for the investigation of iron deficiency was associated with a significant increase in the use of ferritin testing (Figure 4). These are 3 examples in which passive diffusion of physician guidelines were associated with changed behavior in their intended audience.

However, guidelines were not always effective in modifying utilization. There are several reasons why passive dissemination of physician guidelines may not be effective in changing behavior. Not all physicians may receive them, read them, or agree with them. Also, the information within the guidelines may have been previously disseminated and assimilated by physicians responsive to change. Compared with other interventions for altering physician practice, passive guideline dissemination is rather weak.1622 It is possible that some of the guidelines in this study were successful because they were extremely succinct. In addition, they were disseminated by a body (ie, the laboratories themselves) that would not usually supply information that could decrease utilization. Therefore, physicians may have considered the advice to be less biased.

Each time laboratory test options were removed from the laboratory requisition form, laboratory utilization decreased significantly. This finding was observed by Zaat et al,9 who provided 47 Dutch family physicians with a modified form listing fewer test options and compared their laboratory utilization with that of 28 control physicians who used the standard form. The intervention group decreased utilization while the control physicians remained constant. Requisition form modifications are more likely than guidelines dissemination to be effective because changes to the laboratory requisition form reached all physicians. Also, physician habits may drive laboratory requisition patterns. If little thought goes into ordering the test, physician behavior should be easily modified by simple interventions (such as modifying the requisition form) that interrupt a physician's habitual behavior pattern. Lundberg23 noted that, as diners rarely order meals not listed on a restaurant's menu, physicians are less likely to order tests that are not listed on the requisition form.

It is not surprising that all policy changes were associated with significant decreases in laboratory utilization, since they affected test funding and utilization was measured by claims to the health care plan. Other studies have shown policy changes to be effective in changing test volume.7,12

This study has several notable strengths. First, we measured laboratory utilization using payment claims to the single insurance plan within a universal health care system, making this a truly population-based study. Second, age and sex standardization of the utilization rates ensured that changes were not due to alterations in population size or demographics. Finally, time-series modeling allowed us to accurately measure associations between each intervention and changes in laboratory use. The time-series models also allowed cost savings to be estimated.

Several limitations should be noted. First, this study measured associations between the interventions and changes to laboratory utilization. It is possible that events other than the interventions were responsible for utilization changes. Second, we did not determine the appropriateness of laboratory utilization prior to or following the interventions.2 Although the necessity of laboratory tests whose rates were so easily modified is questionable, decreased utilization does not necessarily translate to increased appropriateness. Third, we were unable to determine how laboratory services provided by hospital laboratories were changed by these interventions. Because most patients serviced by these laboratories are hospitalized, they are sicker and are more likely to be treated by physicians in training or specialists. Also, the OHIP laboratory requisition form is not used in hospitals. Therefore, this study's results may not apply to the hospital setting. Fourth, because our study was confined to 1 province in Canada, we cannot be certain that the interventions described would be effective elsewhere. However, given the large sample size of our study and suggestions that inappropriate laboratory utilization has been found in many centers,2 we believe the results are likely generalizable to other health care systems. Finally, many of the interventions were introduced at the same time, thereby making it difficult to determine which intervention was most successful in changing laboratory test ordering behavior. However, this limitation is less concerning because combining complementary interventions has been suggested as the optimal approach to changing physician behavior.16,17

With multiple approaches, it was possible to change the behavior of physicians through guidelines, laboratories through policy, and how the 2 communicate through the laboratory requisition form. This study demonstrates that focusing these 3 methods on low-cost but commonly used health care technologies can result in notable savings for the health care system.24

References
1.
Leese B. Is There Too Much Laboratory Testing? York, England: University of York; 1991:79.
2.
van Walraven C, Naylor CD. Do we know what inappropriate laboratory utilization is?  JAMA.1998;280:550-558.
3.
Axt-Adam P, van der Wouden JC, van der Does E. Influencing behavior of physicians ordering laboratory tests.  Med Care.1993;31:784-794.
4.
Menon D, Hommerhalder S. Influence of Educational Interventions on the Test Ordering Patterns of Physicians. Ottawa, Ontario: Canadian Coordinating Office for Health Technology Assessment; 1992.
5.
Davidoff F, Goodspeed R, Clive J. Changing test ordering behavior.  Med Care.1989;27:45-58.
6.
Thompson RS, Kirz HL, Gold RA. Changes in physician behavior and cost savings associated with organizational recommendations on the use of "routine" chest X rays and multichannel blood tests.  Prev Med.1983;12:385-396.
7.
Mozes B, Lubin D, Modan B, Ben-Bassat I, Gitel SN, Halkin H. Evaluation of an intervention aimed at reducing inappropriate use of preoperative blood coagulation tests.  Arch Intern Med.1989;149:1836-1838.
8.
Wong ET, McCarron MM, Shaw Jr ST. Ordering of laboratory tests in a teaching hospital: can it be improved?  JAMA.1983;249:3076-3080.
9.
Zaat JO, van Eijk JT, Bonte HA. Laboratory test form design influences test ordering by general practitioners in the Netherlands.  Med Care.1992;30:189-198.
10.
Durand-Zaleski I, Rymer JC, Roudot-Thoraval F, Revuz J, Rosa J. Reducing unnecessary laboratory use with new test request form: example of tumour markers.  Lancet.1993;342:150-153.
11.
Novich M, Gillis L, Tauber AI. The laboratory test justified.  Am J Clin Pathol.1985;84:756-759.
12.
Dixon RH, Laszlo J. Utilization of clinical chemistry services by medical house staff: an analysis.  Arch Intern Med.1974;134:1064-1067.
13.
Ohkubo A. Reducing unnecessary laboratory use through education and regulation.  Rinsho Byori.1995;43:647-652.
14.
 Salient Features of the Laboratory Industry in Ontario, 1992-93.  Toronto: Ontario Ministry of Health; 1995.
15.
MacDonald D, From G, Luxton AGW, Naylor CD.for the Working Party on Testing Strategies in Thyroid Disease.  Report of the Working Party on Testing Strategies in Thyroid Disease. Toronto: Ontario Ministry of Health; 1992.
16.
Davis DA, Taylor-Vaisey A. Translating guidelines into practice.  CMAJ.1997;157:408-416.
17.
Oxman AD, Thomson MA, Davis DA, Haynes RB. No magic bullets.  CMAJ.1995;153:1423-1431.
18.
Avorn J, Soumerai SB. Improving drug-therapy decisions through educational outreach.  N Engl J Med.1983;308:1457-1463.
19.
Cohen SJ, Christen AG, Katz BP.  et al.  Counseling medical and dental patients about cigarette smoking.  Am J Public Health.1987;77:313-316.
20.
Everett GD, deBlois CS, Chang PF, Holets T. Effect of cost education, cost audits, and faculty chart review on the use of laboratory services.  Arch Intern Med.1983;143:942-944.
21.
Putnam RW, Curry L. Physicians' participation in establishing criteria for hypertension management in the office.  CMAJ.1989;140:806-809.
22.
Sibley JC, Sackett DL, Neufeld V, Gerrard B, Rudnick KV, Fraser W. A randomized trial of continuing medical education.  N Engl J Med.1982;306:511-515.
23.
Lundberg GD. Laboratory request forms (menus) that guide and teach.  JAMA.1983;249:3075.
24.
Moloney TW, Rogers DE. Medical technology—a different view of the contentious debate over costs.  N Engl J Med.1979;301:1413-1419.
×