Base Rate fallacy. The gist: consider a yes/no test that has a small chance of being wrong (false positive and false negative). Examples: spam filter, virus fi

Base Rate fallacy. The gist: consider a yes/no test that has a small chance of being wrong (false positive and false negative). Examples: spam filter, virus filter, pregnancy test, testing for another illness, etc. How well it performs in reality depends enormously on the real ratio of yes vs. no. Example: Spam filter is 99% correct, 1% false positive and equally 1% false negative. That sounds good! And it is - IF your spam vs ham rate is 50-50. If, however, 99% of mails are spam, and you use this filter on 10,000 mails, then: - 9900 are spam; 100 ham - 1% of ham = 1 mail ends up in spamfolder - 1% of spam = 99 mails end up in inbox. So your inbox now contains 99 ham mails and 99 spam - an unacceptable 50-50 ratio, even though your using a superduper spam filter. Changing the base rate of spam affects the outcome drastically. Obviously, 99% ham and only 1% spam flips the numbers around: inbox is fine, but 50% of your spamfolder isn't spam. Also not really okay -- and that with exactly the same filter, just a different base rate. TL;DR: actual effectiveness of tests which have false positives/negatives is heavily skewed by the frequency of occurrence of whatever you're testing for.