[Skip to Content]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address 18.207.136.184. Please contact the publisher to request reinstatement.
[Skip to Content Landing]
Figure.
Distribution of Responses to Survey Question Provided in the Article Text
Distribution of Responses to Survey Question Provided in the Article Text

Of 61 respondents, 14 provided the correct answer of 2%. The most common answer was 95%, provided by 27 of 61 respondents. The median answer was 66%, which is 33 times larger than the true answer.

Table.  
Survey Respondentsa
Survey Respondentsa
1.
Casscells  W, Schoenberger  A, Graboys  TB.  Interpretation by physicians of clinical laboratory results.  N Engl J Med. 1978;299(18):999-1001.PubMedGoogle ScholarCrossref
2.
Berwick  DM, Fineberg  HV, Weinstein  MC.  When doctors meet numbers.  Am J Med. 1981;71(6):991-998.PubMedGoogle ScholarCrossref
3.
Elstein  AS.  Heuristics and biases: selected errors in clinical reasoning.  Acad Med. 1999;74(7):791-794.PubMedGoogle ScholarCrossref
4.
Association of American Medical Colleges. MR5: 5th Comprehensive Review of the Medical College Admission Test® (MCAT®). https://www.aamc.org/initiatives/mr5/. Accessed October 9, 2013.
5.
Association of American Medical Colleges, Howard Hughes Medical Institute.  Scientific Foundations for Future Physicians.2009.https://www.aamc.org/download/271072/data/scientificfoundationsforfuturephysicians.pdf. Accessed March 6, 2014.
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    3 Comments for this article
    EXPAND ALL
    The authors are not entirely correct in their solution
    Paul Gerrard, MD | Spaulding Rehabilitation Hospital
    The authors are actually wrong about the 2% correct answer. The correct answer to the math problem that the authors presented is actually incalculable.Positive Predictive Value = True Positives / Number of Positive Test Results.The math problem in the study give the false positive rate and the disease prevalence. However, unless I am missing something, it doesn't give the number of those who truly have the disease who will have a positive test result (True positive). Without this, an accurate answer cannot be calculated.To come up with the 2% calculation, the authors made an additional assumption neither stated in the problem nor clearly implied: They had to assume that there is 1 true positive in each 1,000 person sample. (Which will yield 1/51 = 1.96%)It is a bit problematic to say that people got this question wrong, when a correct answer is not calculable without making an additional assumption. Perhaps what is most telling about this article is not that so many subjects got the problem \"wrong,\" but that the authors managed to get it published in one of the most prestigious medical journals despite the fact that a \"right\" answer was not calculable.
    CONFLICT OF INTEREST: None Reported
    READ MORE
    Authors are correct and this is a frequent error
    Dave Curtis | University College London
    Actually, the authors do say that the test is assumed to be perfectly sensitive so they have not made the error Paul Gerrard accuses them of.I have seen this same error even in high profile scientific publications. In a news article in the Lancet entitled \"At-home HIV test poses dilemmas and opportunities\" the original version reported the false positive rate as if it was one minus the specificity. We wrote in and the article has now been corrected so there is no sign of the original error in the online version, just a note to say there has been a correction. (http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(12)61585-2/fulltext)It is difficult enough to have sensible conversations about the advantages and disadvantages of screening and diagnostic tests without such widespread innumeracy among health professionals. I congratulate the authors for highlighting this issue.
    CONFLICT OF INTEREST: None Reported
    READ MORE
    Learning to make assumptions while practising medicine, for there is no certainties in life except births and deaths; we don't need level 1 evidence for everything‏
    Shyan Goh | Sydney, Australia
    The basis of the 1978 paper by Casscells et al was the equation:PPV = (p x Se) / [(p x Se) + (1 - p)(1 - Sp)]where PPV is Positive Predictive Value, p is disease prevalence, Se is test Sensitivity, Sp is test SpecificityRestrictly speaking using the equation from this paper, p = 1/1000, (1- Sp) = False Positive Rate = 5/100If Se is 0, then PPV is 0If Se is 1, then PPV is 100/5095 or 0.0196Regardless of whatever true value of Se is (ie no assumption required), PPV cannot be more than 0.0196However this equation is not commonly used in this format.Whereas in the \"common sense\" reasoning from the actual Casscells et al paper uses the usual equation for PPV beingPPV = TP / (TP + FP)whereby TP is True Positive and FP is False PositiveBased on this equation:1 out of any 1000 people has the diseaseSo 999 people does not have the diseaseFalse Positive Rate of a test for the disease is 5%Of 999 people without the disease, about 0.05 X 999 = just under 50 people would have tested positive on the test.Of 1000 people tested, there can only be maximum of 1 person who is tested positive with the disease (true positive)This (just under) 50 people (false positive) plus 1 person (max possible number of people with the disease to have positive test in a group of 1000 ie maximum possible true positive) means there is a maximum possible just under 51 people who would have tested positive out of 1000 So PPV is TP / (TP + FP) = (maximum possible 1) / (just under 51) = not more than 1/51The issue therefore at hand is how respondents being tested is willing to make some assumption and extrapolation within reason using what is known as possible.The trouble is that not many of us are taught to make some reasonable assumption nowsdays given the direction of medical litigation is going plus the fanatical (and sometimes senseless) call to demand for evidence for every thing we do.The fact is, it IS safe to assume, for example, everyone who jump off the top of a skyscraper, is going to die and does not require level 1 evidence base to support that assumption.Anyone who does survive this fall is a very very lucky person as much luck being winning the grand prize in the biggest El Gordo lottery.
    CONFLICT OF INTEREST: None Reported
    READ MORE
    Research Letter
    June 2014

    Medicine’s Uncomfortable Relationship With Math: Calculating Positive Predictive Value

    Author Affiliations
    • 1Harvard-MIT Health Sciences and Technology, Cambridge, Massachusetts
    • 2Center for Biomedical Informatics, Harvard Medical School, Boston, Massachusetts
    • 3The Eli and Edythe L Broad Institute of MIT and Harvard, Cambridge, Massachusetts
    • 4Department of Veterans Affairs (VA) Boston Healthcare System, Harvard Medical School, Boston, Massachusetts
    • 5Merck Medical Information and Innovation, Merck and Company, Boston, Massachusetts
    • 6Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts
    JAMA Intern Med. 2014;174(6):991-993. doi:10.1001/jamainternmed.2014.1059

    In 1978, Casscells et al1 published a small but important study showing that the majority of physicians, house officers, and students overestimated the positive predictive value (PPV) of a laboratory test result using prevalence and false positive rate. Today, interpretation of diagnostic tests is even more critical with the increasing use of medical technology in health care. Accordingly, we replicated the study by Casscells et al1 by asking a convenience sample of physicians, house officers, and students the same question: “If a test to detect a disease whose prevalence is 1/1000 has a false positive rate of 5%, what is the chance that a person found to have a positive result actually has the disease, assuming you know nothing about the person's symptoms or signs?”

    Methods

    During July 2013, we surveyed a convenience sample of 24 attending physicians, 26 house officers, 10 medical students, and 1 retired physician at a Boston‐area hospital, across a wide range of clinical specialties (Table). Assuming a perfectly sensitive test, we calculated that the correct answer is 1.96% and considered “2%,” “1.96%,” or “<2%” correct. 95% Confidence intervals were computed using the exact binomial and 2‐sample proportion functions in R. The requirement for study approval was waived by the institutional review board of Department of Veterans Affairs Boston Healthcare System.

    Results

    Approximately three-quarters of respondents answered the question incorrectly (95% CI, 65% to 87%). In our study, 14 of 61 respondents (23%) gave a correct response, not significantly different from the 11 of 60 correct responses (18%) in the Casscells study (difference, 5%; 95% CI, −11% to 21%). In both studies the most common answer was “95%,” given by 27 of 61 respondents (44%) in our study and 27 of 60 (45%) in the study by Casscells et al1 (Figure). We obtained a range of answers from “0.005%” to “96%,” with a median of 66%, which is 33 times larger than the true answer. In brief explanations of their answers, respondents often knew to compute PPV but accounted for prevalence incorrectly. For example, one attending cardiologist wrote that “PPV does not depend on prevalence,” and a resident wrote “better PPV when prevalence is low.”

    Discussion

    With wider availability of medical technology and diagnostic testing, sound clinical management will increasingly depend on statistical skills. We measured a key facet of statistical reasoning in practicing physicians and trainees: the evaluation of PPV. Understanding PPV is particularly important when screening for unlikely conditions, where even nominally sensitive and specific tests can be diagnostically uninformative. Our results show that the majority of respondents in this single-hospital study could not assess PPV in the described scenario. Moreover, the most common error was a large overestimation of PPV, an error that could have considerable impact on the course of diagnosis and treatment.

    We advocate increased training on evaluating diagnostics in general. Statistical reasoning was recognized to be an important clinical skill over 35 years ago,1-3 and notable initiatives like the Association of American Medical Colleges–Howard Hughes Medical Institute collaboration have developed recommendations to improve the next generation of medical education.4,5 Our results suggest that these efforts, while laudable, could benefit from increased focus on statistical inference. Specifically, we favor revising premedical education standards to incorporate training in statistics in favor of calculus, which is seldom used in clinical practice. In addition, the practical applicability of medical statistics should be demonstrated throughout the continuum of medical training—not just in medical school.

    To make use of these skills, clinicians need access to accurate sensitivity and specificity measures for ordered tests. In addition, we support the use of software integrated into the electronic ordering system that can prevent common errors and point-of-care resources like smartphones that can aid in calculation and test interpretation. The increasing diversity of diagnostic options promises to empower physicians to improve care if medical education can deliver the statistical skills needed to accurately incorporate these options into clinical care.

    Back to top
    Article Information

    Corresponding Author: Sachin H. Jain, MD, Merck Medical Information and Innovation, Merck & Co, 33 Ave Louis Pasteur, M2I2, Boston, MA 02115 (shjain@post.harvard.edu).

    Published Online: April 21, 2014. doi:10.1001/jamainternmed.2014.1059.

    Author Contributions: Messrs Manrai and Bhatia had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Messrs Manrai and Bhatia contributed equally.

    Study concept and design: Manrai, Bhatia, Strymish, Jain.

    Acquisition, analysis, or interpretation of data: Manrai, Bhatia, Kohane.

    Drafting of the manuscript: Manrai, Bhatia.

    Critical revision of the manuscript for important intellectual content: All authors.

    Statistical analysis: Manrai, Bhatia.

    Study supervision: Strymish, Jain.

    Conflict of Interest Disclosures: None reported.

    Additional Contributions: Paul R. Conlin, MD, VA Boston Healthcare System, helped in designing and executing the study.

    References
    1.
    Casscells  W, Schoenberger  A, Graboys  TB.  Interpretation by physicians of clinical laboratory results.  N Engl J Med. 1978;299(18):999-1001.PubMedGoogle ScholarCrossref
    2.
    Berwick  DM, Fineberg  HV, Weinstein  MC.  When doctors meet numbers.  Am J Med. 1981;71(6):991-998.PubMedGoogle ScholarCrossref
    3.
    Elstein  AS.  Heuristics and biases: selected errors in clinical reasoning.  Acad Med. 1999;74(7):791-794.PubMedGoogle ScholarCrossref
    4.
    Association of American Medical Colleges. MR5: 5th Comprehensive Review of the Medical College Admission Test® (MCAT®). https://www.aamc.org/initiatives/mr5/. Accessed October 9, 2013.
    5.
    Association of American Medical Colleges, Howard Hughes Medical Institute.  Scientific Foundations for Future Physicians.2009.https://www.aamc.org/download/271072/data/scientificfoundationsforfuturephysicians.pdf. Accessed March 6, 2014.
    ×