3 ms·

I think we are talking about slightly different meanings of false positive rate. I think you might be talking about the rate relative to all positive trials. I

I think we are talking about slightly different meanings of false positive rate. I think you might be talking about the rate relative to all positive trials. I think I am talking about the rate relative to all trials that don't meet the conditions of the test. So, you survey 100 women and your test tells me that 10 of them have been president, I say that your test has an observed false positive rate of 10%.

M_Mouse 10y ago
You’re correct in regards to terminology; apparently what I was describing is more related to the terms “positive/negative predictive value.” Your definition of false positive rate is the correct one. But I believe my argument is still valid. If we test a group comprised of only former presidents, all positives must be true positives. So our false positive rate will be 0% while our female test set has been observed to have a false positive rate of 10%. This illustrates that the sample set effects the observed false positive rate. As the rate of occurrence of what we are testing for approaches 100% in the sample set, the false positive rate drops to zero. Inversely, as the incidence of what we are testing for in a (sufficiently large) sample set decreases from 100%, the false positive rate will eventually increase to some non-zero percentage unless the test is perfect. So if we only test meth addicts for meth, our false positive rate will be lower then if we test the population at large.
- jsprogrammer 10y ago
  I think that is true, but we can still distinguish between the false positive rate of the testing mechanism vs. the false positive rate of a particular series of tests (where this would depend, somewhat, in the underlying distribution). I'm not sure what the correct terminology is for this distinction, but I still don't believe Bayes' Theorem would come into play.

M_Mouse 10y ago

You’re correct in regards to terminology; apparently what I was describing is more related to the terms “positive/negative predictive value.” Your definition of false positive rate is the correct one. But I believe my argument is still valid. If we test a group comprised of only former presidents, all positives must be true positives. So our false positive rate will be 0% while our female test set has been observed to have a false positive rate of 10%. This illustrates that the sample set effects the observed false positive rate. As the rate of occurrence of what we are testing for approaches 100% in the sample set, the false positive rate drops to zero. Inversely, as the incidence of what we are testing for in a (sufficiently large) sample set decreases from 100%, the false positive rate will eventually increase to some non-zero percentage unless the test is perfect. So if we only test meth addicts for meth, our false positive rate will be lower then if we test the population at large.

jsprogrammer 10y ago
I think that is true, but we can still distinguish between the false positive rate of the testing mechanism vs. the false positive rate of a particular series of tests (where this would depend, somewhat, in the underlying distribution). I'm not sure what the correct terminology is for this distinction, but I still don't believe Bayes' Theorem would come into play.