[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
May 2006

Pediatricians' Clinical Decision MakingResults of 2 Randomized Controlled Trials of Test Performance Characteristics

Author Affiliations

Copyright 2006 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.2006

Arch Pediatr Adolesc Med. 2006;160(5):487-492. doi:10.1001/archpedi.160.5.487

Objective  To determine whether presenting test performance characteristics influences clinical management.

Design  Two questionnaire-based, randomized controlled trials.

Setting  Mailed surveys with 2 clinical vignettes.

Participants  Randomly selected US pediatricians (N=1502).

Intervention  Vignette-specific, randomly assigned test information: no additional information (control), test characteristics (TC), or TC defined. In the pertussis vignette, the TC group received the direct fluorescent antibody test's sensitivity and specificity, and the TC defined group received the same information with definitions. In the urinalysis vignette, the TC group received the false-positive rate of persistent microhematuria in predicting renal disease, and the TC defined group received a definition of this information.

Main Outcome Measures  In the pertussis vignette, diagnostic test choice and management of erythromycin therapy and hospital discharge plans. In the urinalysis vignette, serum laboratory testing and nephrology referral plans.

Results  Six hundred fifty-three participants (49.5% of those eligible) returned completed surveys. In the pertussis vignette, significantly more of the TC (73%) and TC defined (71%) groups ordered the best-performing test than did controls (21%) (P<.001 for both comparisons). Receiving test characteristics did not significantly affect erythromycin therapy or hospital discharge plans (P≥.40). In the urinalysis vignette, the TC defined group referred to nephrology (30%) and checked laboratory tests (88%) significantly more often than did controls (19%, P=.01; 78%, P=.01, respectively), but the TC and control groups' testing and referral plans did not differ significantly (22% vs 19%, P=.36; 75% vs 78%, P=.48, respectively).

Conclusion  Providing test performance characteristics influenced certain clinical decisions, sometimes in unexpected ways.

The Institute of Medicine's report To Err Is Human1,2 prioritized improvement in the quality of health care and increased awareness of its importance. Although improving care can be difficult,3 advances in information technology, such as electronic medical records and computerized provider order entry, have provided useful tools for this task.4,5 Decision support linked to computer provider order entry has been shown to improve many types of care,6 including medication prescribing,7,8 guideline compliance, and diagnostic test ordering.9-14 The health information technology revolution under way in the United States has been influenced by requirements to report quality, incentives to improve billing efficiencies, and financial support from the Department of Health and Human Services.15 It is only a matter of time before health information technology tools become more widely available.16

As outlined in Medical Decision Making,17 physicians approach clinical problems in a stepwise fashion. A list of hypotheses (differential diagnosis) is generated, the hypotheses are tested, the test results evaluated and interpreted, and a course of action is chosen. During the hypothesis-testing phase of medical decision making, appropriate diagnostic test ordering is important, as excessive use of diagnostic tests is an expensive problem in medicine.18 Physicians' discomfort with uncertainty sometimes motivates them to order tests for defensive reasons.19 Interventions targeting physician behavioral factors20 and using quality improvement strategies21 have had some success at improving test ordering. The format used to present test information can independently influence the hypothesis-testing phase. In a randomized controlled trial,22 general practitioners were given a clinical vignette with test results presented in 1 of 3 ways (test result only, test result and test sensitivity and specificity, or test result and the positive likelihood ratio presented in plain language). Physicians who received the test result and the positive likelihood ratio in plain language more accurately estimated the patient's posttest probability of disease.22 Although this study evaluated the hypothesis-testing phase of medical decision making, the influence of the format of presentation of test characteristics on physicians' choice of tests or patient management has not been rigorously tested, to our knowledge. While decision support linked to computerized provider order entry has reduced unnecessary serologic and radiologic tests ordered among adult patients,10,11,14 the influence of providing test performance with test results on pediatricians' subsequent clinical management has not been studied.

Our study examined 2 sets of research questions to address our hypothesis that presenting general pediatricians with information about test accuracy would influence patient management. Our first research question sought to determine whether presenting pediatricians with the sensitivity and specificity of direct fluorescent antibody (DFA) testing for pertussis would influence their initial choice of a diagnostic test and subsequent patient management. We hypothesized that pediatricians receiving test performance characteristics would choose the test with the highest sensitivity and specificity and would choose an appropriate management plan. Our second research question sought to determine whether presenting pediatricians with the high false-positive rate of persistent microhematuria detected on screening urinalysis (96%) would influence their subsequent patient evaluation. We hypothesized that subjects receiving test performance information would be less likely to evaluate the patient further.

Study design

We implemented 2 questionnaire-based, randomized controlled trials in a mailed survey that asked pediatricians to manage 2 clinical vignettes. The questionnaire required approximately 7 minutes to complete. For each of 2 vignettes, we randomized subjects in a factorial design into 1 of 3 vignette-specific test characteristics groups (Figure 1). Therefore, each subject received 1 of 9 versions of the questionnaire. We randomized subjects using a computerized random-number generator. Subjects were blinded to randomization, and allocation was concealed.

Figure 1.
Randomization of subjects. TC indicates pediatricians who received test characteristics alone; TC defined, pediatricians who received test characteristics defined.

Randomization of subjects. TC indicates pediatricians who received test characteristics alone; TC defined, pediatricians who received test characteristics defined.

In February 2002, we mailed potential subjects a 4-page questionnaire, a prepaid return envelope, a dollar bill as a token of appreciation, and a letter stating that participation was voluntary and that responses would be kept confidential. Subjects were given the opportunity to decline to participate. We sent nonresponders a replacement survey at 4-week intervals, for a maximum of 4 mailings per subject, and sent a reminder postcard after the second mailing. We contacted the institutions of nonresponders to confirm that the mailing addresses were correct. The University of Washington Institutional Review Board approved this study.


Our target study population was a random selection of pediatricians practicing general pediatrics in the United States. We recruited our sample from the 2002 American Medical Association Masterfile of licensed physicians in the United States. Compiled for the US Department of Defense, this list is considered a complete collection of physicians' names, addresses, specialty, age, and sex and is not limited to members of the American Medical Association. From among 44 561 pediatricians without listed secondary specialties, 1502 were randomly selected for participation. To power the study to detect a 15% difference in management choices between test performance information groups and assuming a 50% response rate, we chose our sample size to ensure at least 175 participants in each group.

Questionnaire vignettes

We developed the questionnaire and pilot tested it among 22 nonstudy pediatricians. We will refer to the questionnaire's 2 clinical vignettes as “the pertussis vignette” and “the urinalysis vignette.” For each vignette, subjects were presented with a diagnostic test result followed by their randomly assigned, vignette-specific information about test performance. Both vignettes had a control group, a TC group, and a TC defined group. The TC groups were presented information that included the scientific terms sensitivity, specificity, or false-positive rate, with the corresponding numerical values as frequencies (because medical students understand probabilities easier when they are presented as frequencies23). The TC defined groups received the same information defined using nonscientific terminology.

After being provided with test information, subjects were asked to make management decisions about the case. We assumed that subjects would estimate the posttest probability of disease during the hypothesis-testing phase of medical decision-making, before making management decisions. We hypothesized that providing test performance information would aid subjects in their disease probability estimations, because they are needed to make Bayesian calculations of the posttest probability of disease.17

The pertussis vignette presented a 5-month-old girl with perioral cyanosis and a hacking cough; subjects were instructed to assume that the likelihood she had pertussis was 30% (vignette details are available in an appendix from the corresponding author). Subjects were asked to choose one of several diagnostic tests to evaluate her nasopharyngeal secretions for pertussis, including culture, DFA testing, or polymerase chain reaction. Subjects in the control group were presented with no further information and were asked if they “happened to know the sensitivity and specificity” of the 3 tests. The other 2 groups were given the sensitivity and specificity of culture (35% and 100%, respectively), DFA testing (50% and 95%, respectively), and polymerase chain reaction (95% and 99%, respectively) for Bordetella pertussis.24,25 The vignette continued with information that the patient was hospitalized, DFA testing of her sputum was ordered, and treatment with erythromycin was started. Three days later, her symptoms were somewhat improved, and results of the DFA testing were negative for pertussis. Controls received no additional information about this test result, the TC group was presented the sensitivity and specificity of DFA testing (50% and 95%, respectively), while the TC defined group was given the following information: “The pertussis DFA has a sensitivity of 50%, meaning that if 100 patients infected with pertussis were tested with the pertussis DFA, 50 would test positive and 50 would test negative by DFA. Similarly, the pertussis DFA has a specificity of 95%, meaning that if 100 patients who are not infected with pertussis were tested with the DFA for pertussis, 5 would test positive and 95 would test negative.” We then asked subjects to decide whether to continue the patient's treatment with erythromycin and whether to discharge the patient from the hospital.

The urinalysis vignette presented a healthy 5-year-old boy with persistent microhematuria detected by routine screening urinalysis (vignette details are available in an appendix from the corresponding author). Using data obtained from the literature,26-31 we calculated the false-positive rate of persistent microscopic hematuria detected by screening urinalysis to be 96%. The control group received the test result only, with no additional information. The TC group was given the following information: “When screening asymptomatic patients with urinalysis for diseases whose clinical outcome might be improved with early treatment, persistent microscopic hematuria has an estimated false-positive rate of 96%.” The TC defined group was given the following nontechnical explanation of this information: “The literature suggests that, of 100 asymptomatic children who have persistent microscopic hematuria detected by routine urinalysis, 96 will not have significant nephrologic renal disease for which early treatment might improve clinical outcome.” We then asked subjects if they would refer the patient to a pediatric nephrologist or check his serum electrolyte, urea nitrogen, and creatinine levels.

Additional demographic data

To collect information about subjects' training, we asked if they were board certified in pediatrics, a US medical school graduate, and currently in a pediatric residency program. We defined foreign medical school graduate as a graduate of a medical school located outside of the United States.

To collect information about clinical practice, we asked subjects to report the percentage of clinical time spent in general pediatrics and whether they worked in a practice that had 1 or 2 pediatricians, or more. We defined general pediatrician as one who reported spending more than 80% of clinical time in general pediatrics. We defined small primary practice as a primary practice that included 1 or 2 pediatricians.

Main outcome measures

In the pertussis vignette, we assessed subjects' choice of an initial pertussis diagnostic test and whether they continued the patient's treatment with erythromycin and discharged the patient from the hospital. In the urinalysis vignette, we assessed whether subjects referred the patient to a pediatric nephrologist and/or checked the patient's serum electrolyte, urea nitrogen, and creatinine levels.

Statistical analysis

For each outcome, we performed comparisons between the vignette-specific control group and its TC or TC defined group. We conducted χ2 tests to examine the effect of receiving test characteristics on the 5 dichotomous outcomes. If we found statistically significant differences between groups using a subject characteristic, we conducted logistic regression analyses controlling for that characteristic. All analyses were conducted using STATA 8.0 (StataCorp LP, College Station, Tex).

Study participation

We mailed questionnaires to 1502 physicians identified in the American Medical Association Masterfile as pediatricians without subspecialization. Of these, 106 questionnaires (7%) were returned by the post office without forwarding addresses, and 43 physicians (3%) did not meet inclusion criteria. Of the 1353 potentially eligible subjects, 59 (4%) declined to participate and 653 returned completed surveys, leaving 641 potentially eligible nonresponders (Figure 2), for an estimated response rate32 of 49.75% (653/1313). There were no statistically significant differences by age, sex, or test characteristics group between participants and nonresponders.

Figure 2.
Response to survey. TC indicates pediatricians who received test characteristics alone; TC defined, pediatricians who received test characteristics defined.

Response to survey. TC indicates pediatricians who received test characteristics alone; TC defined, pediatricians who received test characteristics defined.

For the pertussis vignette, 202 participants were in the control group, 231 in the TC group, and 220 in the TC defined group. For the urinalysis vignette, 208 participants were in the control group, 231 in the TC group, and 214 in the TC defined group (Table 1). The only significant difference in participant characteristics by randomization groups was by sex for the urinalysis vignette (P = .02) (Table 1).

Table 1. 
Subject Characteristics by Randomization Groups*
Subject Characteristics by Randomization Groups*
Pertussis vignette

Providing participants with the sensitivity and specificity of the 3 tests influenced which test they chose, as significantly more of the TC (73%) and TC defined (71%) groups chose the polymerase chain reaction test (the only test for which sensitivity and specificity were ≥95%) than did controls (21%, P<.001 in both comparisons) (Table 2). Among the controls, 45% reported that they knew the sensitivity and specificity of pertussis culture, 34% knew the values for DFA testing, and 18% knew the values for polymerase chain reaction.

Table 2. 
Effect of Providing Sensitivity and Specificity of the Bordetella pertussis DFA on Pediatricians' Patient Management
Effect of Providing Sensitivity and Specificity of the Bordetella pertussis DFA on Pediatricians' Patient Management

The vast majority of participants chose to continue the patient's erythromycin therapy, and receiving test characteristics had no significant influence on these decisions (90% of the TC group vs 92% of controls, P = .55; and 94% of the TC defined group vs 92% of controls, P = .40). Similarly, most participants chose to discharge the patient from the hospital, and receiving test characteristics had no significant influence on these decisions (86% of the TC group vs 87% of controls, P = .79; and 87% of the TC defined group vs 87% of controls, P = .99) (Table 2).

Urinalysis vignette

Significantly more participants in the TC defined group than controls chose to refer the patient to a nephrologist (30% vs 19%, P = .01), while there was no significant difference in referral between the TC group and the controls (22% vs 19%, P = .36). Significantly more subjects in the TC defined group than controls decided to check the patient's serum electrolyte, urea nitrogen, and creatinine levels (88% vs 78%, P = .01), while there was no significant difference in laboratory test ordering between the TC group and the controls (75% vs 78%, P = .48) (Table 3). Controlling for sex in logistic regression analyses had little effect on any of these results.

Table 3. 
Effect of Providing False-Positive Rate of Persistent Microhematuria Found on Screening Urinalysis on Pediatricians' Subsequent Patient Evaluation
Effect of Providing False-Positive Rate of Persistent Microhematuria Found on Screening Urinalysis on Pediatricians' Subsequent Patient Evaluation

Providing laboratory test performance characteristics influenced pediatricians' clinical decision making, but to different degrees. In the pertussis vignette, few participants reported familiarity with the performance characteristics of commonly used tests. Providing the sensitivity and specificity of diagnostic tests for pertussis dramatically increased the proportion of participants who chose the best-performing test, but it did not influence their clinical management decisions. The subjects' posttest probability of pertussis was perhaps higher than their treatment threshold. On the other hand, it is possible that their decisions to continue erythromycin therapy were primarily influenced by its low cost or favorable adverse effect profile. In the urinalysis vignette, receiving test characteristics alone did not significantly influence participants' decisions to refer the patient or to conduct serum laboratory tests, but receiving test characteristics defined increased the frequency of both evaluations. The direction of this effect contradicted our hypothesis, as we expected that presenting pediatricians with the high false-positive rate of persistent microhematuria would decrease their subsequent workup of the patient's hematuria. The increased evaluation of microhematuria may have been in part due to the “certainty effect,”33 as presenting the nonzero probability of renal disease may have instilled concern. This result supports others' findings that false-positive laboratory results lead to increased resource use.34

The format of the test performance information influenced practice to different degrees, as patient management after receiving test characteristics defined differed significantly from that controls for 3 of the 5 measures, while receiving the test characteristics alone significantly affected only 1 measure. Perhaps participants in the TC group did not understand the scientific terminology that was used; the TC defined group was given the same information described without using scientific language. Presenting the test characteristics defined might be difficult to implement using computerized provider order entry, because the time required to read lengthy decision support might limit the influence of the intervention. The test characteristics alone information was more concise, but it did not significantly influence most management decisions.

Several limitations to this study warrant comment. First, physicians who returned completed surveys may be different from those who did not. However, there were no significant differences in age or sex between responders and nonresponders, and our response rate was similar to the mean response rates in published studies35,36 of physicians. Second, decisions made by participants in our vignettes may not represent their actual clinical practice. However, vignettes have been shown to be a valid tool for measuring the quality of clinical practice,37 and our primary comparison was between randomized groups. Third, in retrospect it might have been preferable to avoid the use of the term false-positive rate presented to the TC group in the urinalysis vignette. Although some have used this term to refer to 1 minus predictive value positive (as our survey did),38 it also has been used to refer to 1 minus specificity.17,39 A false-positive rate would be termed a false-alarm rate in other disciplines.40 An occasional participant may have been confused by the potential ambiguity of this term. However, we believe that an interpretation of a false-positive rate of 96% as 1 minus specificity would be implausible because it would imply a specificity of 4%, a value that would essentially render any such test useless for clinical diagnostic purposes.

Providing diagnostic test performance characteristics sometimes influenced pediatricians' patient management in unexpected ways. Presenting a representative sample of US pediatricians with the same information in different formats influenced their clinical decisions to different degrees. Future research should determine the format for presenting test characteristics that maximally influences physicians' clinical decision making. In addition, interviews should be performed among pediatricians who do not respond to decision support as expected in an effort to understand why sometimes this information is not influential. In light of the substantial investment that the Department of Health and Human Services recently made to promote the use of health information technology in the United States,15 our findings reinforce the importance of verifying that point-of-care decision support has the desired effect.

Back to top
Article Information

Correspondence: Colin M. Sox, MD, MS, Center for Child Health Care Studies, Department of Ambulatory Care and Prevention, Harvard Medical School, and Harvard Pilgrim Health Care, Inc, 133 Brookline Ave, Sixth Floor, Boston, MA 02215 (colin_sox@hms.harvard.edu).

Accepted for Publication: October 27, 2005.

Funding/Support: This study was supported in part by the Nesholm Family Foundation, Seattle, and by the Clinical Scholars Program (Dr Sox) and the Generalist Physician Faculty Scholars Program (Dr Christakis) of The Robert Wood Johnson Foundation, Princeton, NJ.

Disclaimer: The opinions are those of the authors and do not necessarily represent the views of The Robert Wood Johnson Foundation.

Acknowledgment: We acknowledge the helpful feedback received on this project from Alison Galbraith, MD, MPH, and from Hal Sox, MD. We also acknowledge the assistance of the Clinical Scholars Program work-in-progress seminar at the University of Washington.

Kohn  LTCorrigan  JMDonaldson  MS To Err Is Human: Building a Safer Health System.  Washington, DC National Academy Press1999;
Leape  LLBerwick  DMBates  DW What practices will most improve safety?  JAMA 2002;288501- 507PubMedGoogle ScholarCrossref
Chassin  MR Improving the quality of care  N Engl J Med 1996;3351060- 1064PubMedGoogle ScholarCrossref
Bates  DWGawande  AA Improving safety with information technology  N Engl J Med 2003;3482526- 2534PubMedGoogle ScholarCrossref
Bates  DW The quality case for information technology in healthcare  BMC Med Inform Decis Mak 2002;2e7Google ScholarCrossref
Bates  DWKuperman  GJWang  S  et al.  Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality  J Am Med Inform Assoc 2003;10523- 530PubMedGoogle ScholarCrossref
Chertow  GMLee  JKupermann  GJ  et al.  Guided medication dosing for inpatients with renal insufficiency  JAMA 2001;2862839- 2844PubMedGoogle ScholarCrossref
Christakis  DAZimmerman  FJWright  JAGarrison  MMRivara  FPDavis  RL A randomized controlled trial of point-of-care evidence to improve the antibiotic prescribing practices for otitis media in children  Pediatrics 2001;107e15Accessed June 1, 2005PubMedGoogle ScholarCrossref
Tierney  WMMcDonald  CJMartin  DKRogers  MP Computerized display of past test results: effect on outpatient testing  Ann Intern Med 1987;107569- 574PubMedGoogle ScholarCrossref
Harpole  LHKhorasani  RFiskio  JKuperman  GJBates  DW Automated evidence-based critiquing of orders for abdominal radiographs: impact on utilization and appropriateness  J Am Med Inform Assoc 1997;4511- 521PubMedGoogle ScholarCrossref
Solomon  DHShmerling  RHSchur  PHLew  RFiskio  JBates  DW A computer based intervention to reduce unnecessary serologic testing  J Rheumatol 1999;262578- 2584PubMedGoogle Scholar
Tierney  WMMcDonald  CJHui  SLMartin  DK Computer predictions of abnormal test results: effects on outpatient testing  JAMA 1988;2591194- 1198PubMedGoogle ScholarCrossref
Tierney  WMMiller  MEMcDonald  CJ The effect on test ordering of informing physicians of the charges of outpatient diagnostic tests  N Engl J Med 1990;3221499- 1504PubMedGoogle ScholarCrossref
Chen  PTanasijevic  MJSchoenenberger  RAFiskio  JKuperman  GJBates  DW A computer-based intervention for improving the appropriateness of antiepileptic drug level monitoring  Am J Clin Pathol 2003;119432- 438PubMedGoogle ScholarCrossref
Agency for Healthcare Research and Quality, HHS awards $139 million to drive adoption of health information technology [press release]  October13 2004;Available athttp://www.ahrq.gov/news/press/pr2004/hhshitpr.htmAccessed January 4, 2005
Committee on Quality of Health Care in America; Institute of Medicine, Using Information Technology: Crossing the Quality Chasm: A New Health System for the 21st Century.  Washington, DC National Academy Press2001;164- 180
Sox  HCBlatt  MAHiggins  MCMarton  KISox  HCed Differential diagnosis  Medical Decision Making. Stoneham, Mass Butterworth Publishers1988;9- 26Google Scholar
Bates  DWBoyle  DLRittenberg  E  et al.  What proportion of common diagnostic tests appear redundant?  Am J Med 1998;104361- 368PubMedGoogle ScholarCrossref
DeKay  MLAsch  DA Is the defensive use of diagnostic tests good for patients, or bad?  Med Decis Making 1998;1819- 28PubMedGoogle ScholarCrossref
Solomon  DHHashimoto  HDaltroy  LLiang  MH Techniques to improve physicians' use of diagnostic tests: a new conceptual framework  JAMA 1998;2802020- 2027PubMedGoogle ScholarCrossref
Verstappen  WHvan der Weijden  TSijbrandij  J  et al.  Effect of a practice-based strategy on test ordering performance of primary care physicians: a randomized trial  JAMA 2003;2892407- 2412PubMedGoogle ScholarCrossref
Steurer  JFischer  JEBachmann  LMKoller  MRiet  GT Communicating accuracy of tests to general practitioners: a controlled trial  BMJ 2002;324824- 826PubMedGoogle ScholarCrossref
Kurzenhauser  SHoffrage  U Teaching Bayesian reasoning: an evaluation of a classroom tutorial for medical students  Med Teach 2002;24516- 521PubMedGoogle ScholarCrossref
Tilley  PAKanchana  MVKnight  IBlondeau  JAntonishyn  NDeneer  H Detection of Bordetella pertussis in a clinical laboratory by culture, polymerase chain reaction, and direct fluorescent antibody staining; accuracy, and cost  Diagn Microbiol Infect Dis 2000;3717- 23PubMedGoogle ScholarCrossref
Loeffelholz  MJThompson  CJLong  KSGilchrist  MJ Comparison of PCR, culture, and direct fluorescent-antibody testing for detection of Bordetella pertussis  J Clin Microbiol 1999;372872- 2876PubMedGoogle Scholar
Vehaskari  VMRapola  JKoskimies  OSavilahti  EVilska  JHallman  N Microscopic hematuria in school children: epidemiology and clinicopathologic evaluation  J Pediatr 1979;95676- 684PubMedGoogle ScholarCrossref
Feld  LGMeyers  KEKaplan  BSStapleton  FB Limited evaluation of microscopic hematuria in pediatrics  Pediatrics 1998;102e42PubMedGoogle ScholarCrossref
Hisano  SKwano  MHatae  K  et al.  Asymptomatic isolated microhaematuria: natural history of 136 children  Pediatr Nephrol 1991;5578- 581PubMedGoogle ScholarCrossref
Ito  KKawaguchi  HHattori  M Screening for proteinuria and hematuria in school children: is it possible to reduce the incidence of chronic renal failure in children and adolescents?  Acta Paediatr Jpn 1990;32710- 715PubMedGoogle ScholarCrossref
Murakami  MYamamoto  HUeda  YMurakani  DYamauchi  K Urinary screening of elementary and junior high-school children over a 13-year period in Tokyo  Pediatr Nephrol 1991;550- 53PubMedGoogle ScholarCrossref
Gauthier  BTrachtman  HFrank  RValderrama  E Familial thin basement membrane nephropathy in children with asymptomatic microhematuria  Nephron 1989;51502- 508PubMedGoogle ScholarCrossref
Council of American Survey Research Organizations, On the Definition of Response Rates: A Special Report From the CASRO Task Force on Completion Rates.  Port Jefferson, NY Council of American Survey Research Organizations1982;
Kahneman  DTversky  AKahneman  DedSlovic  PedTversky  Aed Choices, values and frames  Choices, Values and Frames. 3rd ed. Cambridge, England Cambridge University Press2000;1- 16Google Scholar
Bates  DWGoldman  LLee  TH Contaminate blood cultures and resource utilization: the true consequences of false-positive results  JAMA 1991;265365- 369PubMedGoogle ScholarCrossref
Asch  DAJedrzeiwski  MKChristakis  NA Response rates to mail surveys published in medical journals  J Clin Epidemiol 1997;501129- 1136PubMedGoogle ScholarCrossref
Cummings  SMSavitz  LAKonrad  TR Reporting response rates to mailed physician questionnaires  Health Serv Res 2001;351347- 1355PubMedGoogle Scholar
Peabody  JWLuck  JGlassman  P  et al.  Measuring the quality of physician practice by using clinical vignettes: a prospective validation study  Ann Intern Med 2004;141771- 780PubMedGoogle ScholarCrossref
Fleiss  JL Statistical Methods for Rates and Proportions. 2nd ed. New York, NY John Wiley & Sons Inc1981;
Rembold  CMWatson  D Posttest probability calculation by weights: a simple form of Bayes' theorem  Ann Intern Med 1988;108115- 120PubMedGoogle ScholarCrossref
Olson  RH On the use of Bayes' theorem in estimating false alarm rate  Mon Weather Rev 1965;19557- 558Google ScholarCrossref