5 ms·

One thing this article touches, but does not give enough attention to (maybe because it is so counter-intuitive and hard to explain) is the false positive parad

by aylons 11y ago

One thing this article touches, but does not give enough attention to (maybe because it is so counter-intuitive and hard to explain) is the false positive paradox. To put in perspective, you should only do a test if you have good a priori chance of having a positive. Even a test with 1% false-positive rate you give you back much more false positives than real positives if the a priori chance is 0.1%. This is counter intuitive, I'm not good at explaining stuff, so I'll give Doctorow the word, through this excerpt in Wikipedia: http://en.wikipedia.org/wiki/False_positive_paradox http://en.wikipedia.org/wiki/False_positive_paradox Probabilistics and statistics are hard, very important an counter-intuitive. Med and pre-med, and even high schools really should teach much more math than they do right now.

skywhopper 11y ago
I was best able to understand it by running the numbers. Take a condition with a 0.1% occurrence rate in the population: then 1,000 out of every million people have this condition. Then, say you develop a test for the condition that's "99% accurate". Sounds great! Out of the thousand people with the condition, 99%, or 990, will get a positive test result! Unfortunately, of the people without the condition, 1%, or 9,990, will get a positive test result. So if you're one of the 10,980 people who get a positive test result from this "99% accurate" test, it turns out you still only have a 9% chance of actually having the condition.
jessriedel 11y ago
Everything you wrote is reasonable except this: > To put in perspective, you should only do a test if you have good a priori chance of having a positive. There are several reasons to run a test that yields a small posteriori probability even after a positive result (i.e., a test such that most positives are false positive). The most common reason is that this test is much cheaper (or has fewer risks, etc.) than a more accurate test. The protocol, then, is to run the cheap test and then run the expensive test if you get an initial positive result. The only time you end up believing it's a true positive is when the expensive test returns positive. This is fairly common in medicine, for example the syphilis test: http://www.webmd.com/sexual-conditions/syphilis-tests http://www.webmd.com/sexual-conditions/syphilis-tests
- valleyer 11y ago
  Or, if the false positives are probabilistically independent, just run the same test again. If your test gives back 1% false positives, running it twice gets you down to .01%, three times gives you one in a million.
  - jessriedel 11y ago
    Unfortunately, this is essentially never the case. The reason is that if the outcomes were probabilistically independent, then you would just declare a standard protocol of always running the test N times, and call that the test. It's safe to assume that such trivial optimizations have already been done for medical tests.
    - valleyer 11y ago
      Fair enough. What is an example of a test that has false positives that are not independent (and why)? Curious now.
      - jessriedel 11y ago
        The common syphilis test looks for antibodies that are produced in response to the bacteria. However, the body may produce those antibodies (on a continuing basis) for other reasons. If you find these antibodies the first time in a given person, you'll keep finding them when you re-test.
doctorCoder 11y ago
Counterpoint here, the false positive paradox won't surprise many physicians. It stems from concepts that are well-known among my colleagues and have been tested on each of my licensure exams. We apply them at least 10s of times each day. You can read more about those concepts here- (Note: "a priori" === "pretest probability" in MD speak) http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2636062/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2636062/ https://en.wikipedia.org/wiki/Likelihood_ratios_in_diagnostic_testing https://en.wikipedia.org/wiki/Likelihood_ratios_in_diagnosti...
- shas3 11y ago
  I will add my anecdote to support this comment. Being an EE with probability-heavy background, I tried to give a "you guys don't use prior probabilities" sermon to a physician and his physician-in-training daughter. I was made to eat my words by their reply, which was essentially that it is something they are all taught and use all the time in assessing risks, interpreting results, etc. The one time I had to pay $500 for some tests not covered by insurance, but egregiously wrong in the context of my ailment, the referral was from a nurse practitioner. Two things: 1. Perhaps doctors don't uniformly apply their training in interpreting test results and analyzing risks? (It is after all, quite common in all professions to forget some parts of one's training) 2. In addition to doctors, there are also technicians and nurse practitioners, who some times may not have the rigorous training that one associates with med school.
jberryman 11y ago
Software people might be interested in "The Bloom Paradox" for a case study in this kind of thinking.
- sukilot 11y ago
  That appears to be from a 2012 paper, even though the result it trivial undergraduate statistics material, known for decades?
sukilot 11y ago
Canonical source: http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 http://journals.plos.org/plosmedicine/article?id=10.1371/jou...

by aylons 11y ago

skywhopper 11y ago

I was best able to understand it by running the numbers. Take a condition with a 0.1% occurrence rate in the population: then 1,000 out of every million people have this condition. Then, say you develop a test for the condition that's "99% accurate". Sounds great! Out of the thousand people with the condition, 99%, or 990, will get a positive test result! Unfortunately, of the people without the condition, 1%, or 9,990, will get a positive test result. So if you're one of the 10,980 people who get a positive test result from this "99% accurate" test, it turns out you still only have a 9% chance of actually having the condition.

jessriedel 11y ago

Everything you wrote is reasonable except this: > To put in perspective, you should only do a test if you have good a priori chance of having a positive. There are several reasons to run a test that yields a small posteriori probability even after a positive result (i.e., a test such that most positives are false positive). The most common reason is that this test is much cheaper (or has fewer risks, etc.) than a more accurate test. The protocol, then, is to run the cheap test and then run the expensive test if you get an initial positive result. The only time you end up believing it's a true positive is when the expensive test returns positive. This is fairly common in medicine, for example the syphilis test: http://www.webmd.com/sexual-conditions/syphilis-tests http://www.webmd.com/sexual-conditions/syphilis-tests

valleyer 11y ago
Or, if the false positives are probabilistically independent, just run the same test again. If your test gives back 1% false positives, running it twice gets you down to .01%, three times gives you one in a million.
- jessriedel 11y ago
  Unfortunately, this is essentially never the case. The reason is that if the outcomes were probabilistically independent, then you would just declare a standard protocol of always running the test N times, and call that the test. It's safe to assume that such trivial optimizations have already been done for medical tests.
  - valleyer 11y ago
    Fair enough. What is an example of a test that has false positives that are not independent (and why)? Curious now.
    - jessriedel 11y ago
      The common syphilis test looks for antibodies that are produced in response to the bacteria. However, the body may produce those antibodies (on a continuing basis) for other reasons. If you find these antibodies the first time in a given person, you'll keep finding them when you re-test.

doctorCoder 11y ago

Counterpoint here, the false positive paradox won't surprise many physicians. It stems from concepts that are well-known among my colleagues and have been tested on each of my licensure exams. We apply them at least 10s of times each day. You can read more about those concepts here- (Note: "a priori" === "pretest probability" in MD speak) http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2636062/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2636062/ https://en.wikipedia.org/wiki/Likelihood_ratios_in_diagnostic_testing https://en.wikipedia.org/wiki/Likelihood_ratios_in_diagnosti...

shas3 11y ago
I will add my anecdote to support this comment. Being an EE with probability-heavy background, I tried to give a "you guys don't use prior probabilities" sermon to a physician and his physician-in-training daughter. I was made to eat my words by their reply, which was essentially that it is something they are all taught and use all the time in assessing risks, interpreting results, etc. The one time I had to pay $500 for some tests not covered by insurance, but egregiously wrong in the context of my ailment, the referral was from a nurse practitioner. Two things: 1. Perhaps doctors don't uniformly apply their training in interpreting test results and analyzing risks? (It is after all, quite common in all professions to forget some parts of one's training) 2. In addition to doctors, there are also technicians and nurse practitioners, who some times may not have the rigorous training that one associates with med school.

jberryman 11y ago

Software people might be interested in "The Bloom Paradox" for a case study in this kind of thinking.

sukilot 11y ago
That appears to be from a 2012 paper, even though the result it trivial undergraduate statistics material, known for decades?

sukilot 11y ago

Canonical source: http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 http://journals.plos.org/plosmedicine/article?id=10.1371/jou...