11 ms·

I was recently trying to explain Bayesian logic to a friend, and came up with the following analogy. I would be interested to hear feedback on it. --- Imagine

I was recently trying to explain Bayesian logic to a friend, and came up with the following analogy. I would be interested to hear feedback on it. --- Imagine everyone in the USA gets sudden amnesia. We want to find out who the President is, but no one can remember. A scientist comes up with a test to determine if someone is the President. If they are the President, there is a 100% chance the test will say they are the President and a 0% chance the test will say they are not the President. If they are not the President, there is a 99.999% chance the test will say that are not the President, and a 0.001% chance the test will falsely say they are the President. Giving the test to the person sitting in the big chair in the Oval Office is useful, because it's already quite likely this person is the President. If the test is positive for Presidency, it's extremely likely that person is the president. Giving the test to the 10 people nearest the oval office is useful, because it's fairly likely the President is one of these people. A positive result will indicate strongly that that person is the President, and if no-one in that group is actually the President, there's a 99.99% chance the test will say so. Giving the test to the 1000 people in the White House is pretty useful, because it's pretty likely the President is in the White House, and if none of these people are the president, there's still a 99% chance the test will be correct. A positive result for any one person will indicate quite strongly that that person is the President. But giving the test to everyone in America is not very useful at all, because it's very unlikely that any particular person is the President, and we can expect the test will give a positive result for around 3200 people. For any particular person in this group, it's much more likely they're not the President than they are. --- Is this a broadly correct, if non-rigorous, analogy? I realize most HNers will be much more familiar with this stuff than I am, I'm interested chiefly in whether or not I misled my friend.

tlarkworthy 13y ago
yeah its similar to a classic rare disease screening example in text books: http://www.math.hmc.edu/funfacts/ffiles/30002.6.shtml http://www.math.hmc.edu/funfacts/ffiles/30002.6.shtml (sorry its a bit informal, it was first google hit, but I have seen that in real textbooks for sure) Although its a bit confusing associating the prior with geometric proximity to the oval chair.
klodolph 13y ago
Right. And the frequentist version is also useful. If you test one person and the the test is positive, then that person is the president (p=0.00001). If you test a thousand people, and the test is positive for one of them, then that person is the president (p=0.01). So you don't really need Bayesian logic to reason that you should test fewer people if you want a more significant result. (Note I'm not saying you don't need Bayes' Theorem, which everyone uses.) Edit: I think most people on HN get their knowledge of frequentist and bayesian statistics from XKCD #1132. That's sad.
- jey 13y ago
  > So you don't really need Bayesian logic to reason that you should test fewer people if you want a more significant result. That's misleading by the use of the word "significant" which apparently means something in frequentism than it does in normal speech. I certainly wouldn't use "significant" in that way as a non-frequentist, I would instead rephrase what you said as: > So you don't really need Bayesian logic to reason that you should test fewer people if you want more confirmation bias in your result. And that's a statement I can definitely get behind!
  - klodolph 13y ago
    That's a good point. Conversely, when you run a test and get a result which is "not statistically significant", that doesn't mean that the measured effect is not significant. For example, if I test a drug and say "it does not cause cancer (p=0.17)", that's wrong. Yes, the correlation with cancer is not statistically significant, but it is significant in the normal sense of the word.
  - _delirium 13y ago
    It can definitely be used misleadingly, but it's not too out of line with normal scientific usage of the term. The "significant" in "significant figures" is the same: if a number has "11 significant figures", it doesn't mean the 11th digit is significant in the sense of being important or having a big impact, just that the 11th digit is within the measurement precision (as propagated through any subsequent calculations).
tedsanders 13y ago
xkcd's version sounds similar, but simpler: http://xkcd.com/1132/ http://xkcd.com/1132/
- klodolph 13y ago
  That's the most famous comparison between frequentist and bayesian statistics, and it's a shame because the frequentist interpretation depicted in the comic is really a straw man argument. See: http://stats.stackexchange.com/questions/43339/whats-wrong-with-xkcds-frequentists-vs-bayesians-comic http://stats.stackexchange.com/questions/43339/whats-wrong-w...
  - haberman 13y ago
    I'm not sure it's a straw man, there seems to be plenty of examples of people publishing results based on p < 0.05 findings even when it wasn't justified... just like the person mentioned in the Nature article almost did.
    - jamesaguilar 13y ago
      In a world where all statisticians were Bayesian, you'd still see results like this published. It would simply read differently: "We saw something that seems pretty unlikely, care to confirm?"
      - haberman 13y ago
        Perhaps, or perhaps the scientist would try to replicate the results themselves first (as the person in the article did). But even if they published them with the verbiage you mentioned, that would be a better world because reporters would be less likely to turn it into a story with a headline like "Scientists say ...". The prevalence of articles like this erode trust in science, and fill the public's head with misconceptions.
- truantbuick 13y ago
  This has probably been addressed millions of times, but just to give a response to why this is a misleading comic: If you were equally mocking of both Bayesian and Frequentist, the Bayesian would arguably come up with basically the same conclusion. In this scenario where the Frequentist ignores previous data based on human history and our understanding of the lifecycle of the sun based on physics, the Bayesian should do the same if we are fair. Then his prior distribution would likely be 50% belief that the sun would explode (Why favor either outcome? Thus the prior distribution is split down the middle between both outcomes.). With the new information given from the detector, his posterior would be 35/36 probability that the sun has exploded. Maybe the Bayesian approach provides a more explicit way of incorporating previous information, and without that, it's cause for some misuse with a Frequentist approach, but that doesn't mean Frequentists need be fundamentally ignorant in their approach. The comic's good for a laugh. Just don't take it too seriously as a criticism.
joe_the_user 13y ago
OK, So just to get a handle on this stuff, the two problems you have if you do a test and only look at a low p are: 1) Unusual things do occur. If a million people do the same test, it's obvious they'll come up with some wrong values. It's less obvious that a similar number of wrong values will come if a million people do a million different tests with a similar small chance of bogus results. 2) The pattern of results may indeed be unusual but not necessarily in the fashion you think it is. There may be a non-random pattern possessed by the data but may not because of your particular hypothesis but a "this is not random" result may seem to say your hypothesis does explain the data. Does that characterize the problem?
pcrh 13y ago
A big problem with Bayesian statistics (as I see it) it that it not always possible to have any sense of what the prior probability is. Say you are looking for genes that might influence the rate of occurrence a particular disease. There might be genes that influence this rate, or there might not, it could be entirely environmental, or it could be entirely genetic, or something in between. In any case, you go genome-wide studies, and find that certain gene variants occur more often in your diseased population than in your control population. You apply frequentist statistics, using some corrections for multiple hypothesis testing, and get some kind of "significant" result. This gets published in Nature (you lucky thing!). Are your conclusions correct? Do the genes you identified really modify the course of the disease you studied? Bayesian statistics won't give you the answer. The only way to get the answer is to do experimental science, i.e. deliberately modify the gene(s) in question and show that your modifications change the occurrence or course of the disease. Unfortunately, that is not always feasible, for either technical or ethical reasons, so we have to fall back on the poor cousin of experimental science that is population statistics.
- judofyr 13y ago
  > A big problem with Bayesian statistics (as I see it) it that it not always possible to have any sense of what the prior probability is. Reminds me of this quote: (These values are known as priors, which is ironic because Bayesians inevitably pull them out of their posteriors.) http://plover.net/~bonds/cultofbayes.html http://plover.net/~bonds/cultofbayes.html
  - pcrh 13y ago
    Thanks for that, it was entertaining to read.
  - cossatot 13y ago
    If you learn about Bayes from crazy people, what is the likelihood that you will think Bayesian reasoning is always used in crazy ways? Last week I did some Bayesian (!!) work on finding a pdf for the direction of a certain quantity. My priors were... [0,2*pi).
- 13y ago
reyan 13y ago
This is the correct xkcd for this analogy http://xkcd.com/882/ http://xkcd.com/882/, not #1132.
deleted 13y ago
[deleted]
doctorpangloss 13y ago
>A scientist comes up with a test to determine if someone is the President. It's a poor analogy, because it's not clear to people why such a test is "natural." It's not clear how your specific test could be broken in the peculiar way that it would have 99.999% chance to confirm that someone who isn't the president is indeed not the president. And people would get caught up with what you mean by "If they are" and "If they are not," since it's not clear how you would know the error of your test without a real president around to identify. False positives or false negatives are not at all intuitive to people who have never done experimental design. Most people would get stuck at percentages anyhow.

tlarkworthy 13y ago

yeah its similar to a classic rare disease screening example in text books: http://www.math.hmc.edu/funfacts/ffiles/30002.6.shtml http://www.math.hmc.edu/funfacts/ffiles/30002.6.shtml (sorry its a bit informal, it was first google hit, but I have seen that in real textbooks for sure) Although its a bit confusing associating the prior with geometric proximity to the oval chair.

klodolph 13y ago

Right. And the frequentist version is also useful. If you test one person and the the test is positive, then that person is the president (p=0.00001). If you test a thousand people, and the test is positive for one of them, then that person is the president (p=0.01). So you don't really need Bayesian logic to reason that you should test fewer people if you want a more significant result. (Note I'm not saying you don't need Bayes' Theorem, which everyone uses.) Edit: I think most people on HN get their knowledge of frequentist and bayesian statistics from XKCD #1132. That's sad.

jey 13y ago
> So you don't really need Bayesian logic to reason that you should test fewer people if you want a more significant result. That's misleading by the use of the word "significant" which apparently means something in frequentism than it does in normal speech. I certainly wouldn't use "significant" in that way as a non-frequentist, I would instead rephrase what you said as: > So you don't really need Bayesian logic to reason that you should test fewer people if you want more confirmation bias in your result. And that's a statement I can definitely get behind!
- klodolph 13y ago
  That's a good point. Conversely, when you run a test and get a result which is "not statistically significant", that doesn't mean that the measured effect is not significant. For example, if I test a drug and say "it does not cause cancer (p=0.17)", that's wrong. Yes, the correlation with cancer is not statistically significant, but it is significant in the normal sense of the word.
- _delirium 13y ago
  It can definitely be used misleadingly, but it's not too out of line with normal scientific usage of the term. The "significant" in "significant figures" is the same: if a number has "11 significant figures", it doesn't mean the 11th digit is significant in the sense of being important or having a big impact, just that the 11th digit is within the measurement precision (as propagated through any subsequent calculations).

tedsanders 13y ago

xkcd's version sounds similar, but simpler: http://xkcd.com/1132/ http://xkcd.com/1132/

klodolph 13y ago
That's the most famous comparison between frequentist and bayesian statistics, and it's a shame because the frequentist interpretation depicted in the comic is really a straw man argument. See: http://stats.stackexchange.com/questions/43339/whats-wrong-with-xkcds-frequentists-vs-bayesians-comic http://stats.stackexchange.com/questions/43339/whats-wrong-w...
- haberman 13y ago
  I'm not sure it's a straw man, there seems to be plenty of examples of people publishing results based on p < 0.05 findings even when it wasn't justified... just like the person mentioned in the Nature article almost did.
  - jamesaguilar 13y ago
    In a world where all statisticians were Bayesian, you'd still see results like this published. It would simply read differently: "We saw something that seems pretty unlikely, care to confirm?"
    - haberman 13y ago
      Perhaps, or perhaps the scientist would try to replicate the results themselves first (as the person in the article did). But even if they published them with the verbiage you mentioned, that would be a better world because reporters would be less likely to turn it into a story with a headline like "Scientists say ...". The prevalence of articles like this erode trust in science, and fill the public's head with misconceptions.
truantbuick 13y ago
This has probably been addressed millions of times, but just to give a response to why this is a misleading comic: If you were equally mocking of both Bayesian and Frequentist, the Bayesian would arguably come up with basically the same conclusion. In this scenario where the Frequentist ignores previous data based on human history and our understanding of the lifecycle of the sun based on physics, the Bayesian should do the same if we are fair. Then his prior distribution would likely be 50% belief that the sun would explode (Why favor either outcome? Thus the prior distribution is split down the middle between both outcomes.). With the new information given from the detector, his posterior would be 35/36 probability that the sun has exploded. Maybe the Bayesian approach provides a more explicit way of incorporating previous information, and without that, it's cause for some misuse with a Frequentist approach, but that doesn't mean Frequentists need be fundamentally ignorant in their approach. The comic's good for a laugh. Just don't take it too seriously as a criticism.

joe_the_user 13y ago

OK, So just to get a handle on this stuff, the two problems you have if you do a test and only look at a low p are: 1) Unusual things do occur. If a million people do the same test, it's obvious they'll come up with some wrong values. It's less obvious that a similar number of wrong values will come if a million people do a million different tests with a similar small chance of bogus results. 2) The pattern of results may indeed be unusual but not necessarily in the fashion you think it is. There may be a non-random pattern possessed by the data but may not because of your particular hypothesis but a "this is not random" result may seem to say your hypothesis does explain the data. Does that characterize the problem?

pcrh 13y ago

A big problem with Bayesian statistics (as I see it) it that it not always possible to have any sense of what the prior probability is. Say you are looking for genes that might influence the rate of occurrence a particular disease. There might be genes that influence this rate, or there might not, it could be entirely environmental, or it could be entirely genetic, or something in between. In any case, you go genome-wide studies, and find that certain gene variants occur more often in your diseased population than in your control population. You apply frequentist statistics, using some corrections for multiple hypothesis testing, and get some kind of "significant" result. This gets published in Nature (you lucky thing!). Are your conclusions correct? Do the genes you identified really modify the course of the disease you studied? Bayesian statistics won't give you the answer. The only way to get the answer is to do experimental science, i.e. deliberately modify the gene(s) in question and show that your modifications change the occurrence or course of the disease. Unfortunately, that is not always feasible, for either technical or ethical reasons, so we have to fall back on the poor cousin of experimental science that is population statistics.

judofyr 13y ago
> A big problem with Bayesian statistics (as I see it) it that it not always possible to have any sense of what the prior probability is. Reminds me of this quote: (These values are known as priors, which is ironic because Bayesians inevitably pull them out of their posteriors.) http://plover.net/~bonds/cultofbayes.html http://plover.net/~bonds/cultofbayes.html
- pcrh 13y ago
  Thanks for that, it was entertaining to read.
- cossatot 13y ago
  If you learn about Bayes from crazy people, what is the likelihood that you will think Bayesian reasoning is always used in crazy ways? Last week I did some Bayesian (!!) work on finding a pdf for the direction of a certain quantity. My priors were... [0,2*pi).
13y ago

reyan 13y ago

This is the correct xkcd for this analogy http://xkcd.com/882/ http://xkcd.com/882/, not #1132.

deleted 13y ago

[deleted]

doctorpangloss 13y ago

>A scientist comes up with a test to determine if someone is the President. It's a poor analogy, because it's not clear to people why such a test is "natural." It's not clear how your specific test could be broken in the peculiar way that it would have 99.999% chance to confirm that someone who isn't the president is indeed not the president. And people would get caught up with what you mean by "If they are" and "If they are not," since it's not clear how you would know the error of your test without a real president around to identify. False positives or false negatives are not at all intuitive to people who have never done experimental design. Most people would get stuck at percentages anyhow.