6 ms·

Here's a good example of how p values are misleading: Consider a bag containing two coins, one that's fair and one that always shows heads. Pick one coin at ra

by pontus 6y ago

Here's a good example of how p values are misleading: Consider a bag containing two coins, one that's fair and one that always shows heads. Pick one coin at random and flip it ten times. Suppose you see ten heads. Would you be comfortable saying that you have the biases coin? Most likely you would. Formally you'd compute the p value (in this case something like 1/1024 or 2/1024 depending on if you do a one sided or two sided test). You'd compare this with some cutoff, perhaps 0.05 and claim that you can reject the null hypothesis. So far so good. Now let's perform that experiment again but this time instead of having one fair coin and one biased coin in the bag, suppose we have one biases coin and 10^1000 fair coins. Suppose again that you randomly pick a coin from the bag and flip it ten times. Again you see ten heads. Do you think you have the biases coin? Of course not! What's the chance that you just happened to choose the single biased coin instead of one of the 10^1000 fair coins? Practically zero. In this case it's much more likely that the ten heads were a statistical fluke from a fair coin. However, if you compute the p value you still get 1/1024 (or 2/1024 for a two sided test). This example should be reasonably convincing that one needs access to a prior in order to correctly interpret p values. One way of looking at it is that you need to set the threshold appropriately based on the plausibility of the alternate hypothesis. Another way to phrase it is that we actually need to compute the posterior Bayesian probability of the null hypothesis being true. There are tons of other confusing / surprising examples where a naive p-value calculation will be misleading. I wrote about these things here: https://mindbowling.wordpress.com/2016/07/19/p-values/ https://mindbowling.wordpress.com/2016/07/19/p-values/

nickhuh 6y ago
I think this example misleads one's intuition for the following reason: in the proposed scenario, you'd only see the coin come up heads 10 times in a row about 1 in 1024 times you ran the experiment. While your conclusion would likely be incorrect, you almost never run into that scenario. For example, if you conducted a study every week for 20 years, you'd both be extremely prolific and expect to have drawn about one wrong conclusion. The example is a case of an absurd premise (i.e., a fair coin comes up heads 10 times in a row) leading to an absurd conclusion (that the coin is biased). Of course, this is exactly the guarantee the hypothesis test provides: under robust assumptions, you'll draw the wrong conclusion only rarely.
- jbay808 6y ago
  A 1 in 1024 chance is not an absurd premise! Something that has a 1 in 1024 chance of ocurring to a person happens to 7 million people. If you get that evidence, you need to be capable of interpreting it correctly. The OP's example would also worked well enough with even just 5 heads; you'd already pass p < 0.05, despite the actual probability of having picked the biased coin still being minuscule.
- feanaro 6y ago
  I've been playing a board game daily for the last two weeks. As part of this game, a player has to draw one out of six unique cards. In the last five games, I've repeatedly drawn the same card. This is a real example. 1 / (1/6)^5 ≃ 7776
  - MaximumYComb 6y ago
    You can't rely on odds after an event has happened to determine probably. I could shuffle a deck containing 1000 unique cards and then look at their order. It's astronomically low that this order occurs but it did happen.
    - joshuamorton 6y ago
      When discussing conditional probability, this is absolutely what you can do. Let's use your example. Conditional probability is P(A|B), the probability of event A, given that event B was observed. What's the probability that I am a magician, given that I shuffled the deck and when you saw it it was still in new deck order? Now certainly, there is an astronomically small chance that this was observed due to random chance. And yet if you observed this, I'd expect that you would, with relatively high confidence, believe that I stacked the deck.
    - feanaro 6y ago
      I'm not doing anything different than what we are doing when discussing the improbability of a hypothetical 10-heads-in-a-row event.
- 6y ago
tpoacher 6y ago
Except this is not what a p-value is supposed to tell you in the first place. It's supposed to tell you how compatible your observations are with the ASSUMED theory. The minute you use it as an interpretation of a posterior probability (let alone a malformed one), you have violated the assumption on which you rely to interpret it. To use your own example. If you picked a coin from said bag which is mostly full of otherwise unbiased coins, and your hypothesis is "this coin is not biased", a sequence of ten heads would be largely "incompatible" with your hypothesis (as if the hypothesis of unbiasedness was true, you should only expect this result 1/1024 times). It is up to you to then place this result in the context of other prior assumptions etc, and say "yes it's incompatible but still very likely". The p-value does not preclude you from performing a post-hoc analysis of the base-rate if you so wish. It just so happens that the 'assume a hypothesis' approach is a useful one in the absence of all other information. Which is not so different from a Bayesian saying 'I'll use an uninformative prior and let the data speak for itself'. But using it as a test of posterior probability, especially when prior information exists, and saying "ha see it sucks" is not a fair criticism for it (whereas criticising its use as a test of posterior probability is).
- pontus 6y ago
  Yes, that's exactly my point. Saying that it really supposed to be the probability of the data assuming the null hypothesis is true is pretty confusing to most people and often times the gut feeling people keep us that it still somehow tells you how likely it is that the null hypothesis is true. I just like this example because it highlights that there is a distinction between these two probabilities.
- pontus 6y ago
  In fact, if you look at the document I link to above, you'll see the correct Bayesian calculation where we actually compute p(Biased | 10H). In the first case it's pretty high but in the second case it's very small.
  - tpoacher 6y ago
    I 100% agree that p-values are hard to get right (not that Bayesian analysis is necessarily more straightforward, mind you). What I disagree with is the assertion that a) the two scenarios listed above are equivalent, and b) that you cannot incorporate a prior analysis for use with p-values (which is best thought simply as a measures of compatibility of the data with respect to a hypothesis). E.g. in the biased vs unbiased example, if your null hypothesis is "The coin is biased", then it is simply the wrong hypothesis for the problem (or, more specifically, the wrong null/alternative pair, since these two only make sense as a pair; a null by itself is not sufficient to define the assumptions for the problem). If your alternative hypothesis is "ok, it's one of the 10^1000-1 unbiased ones", then your null should have necessarily been "it's the biased coin with 1/10^1000 chance of being chosen", which effectively forces you to apply your prior probabilities on the hypotheses themselves. Once you do that, you will easily see that the compatibility of the data with the alternative hypothesis is much higher, exactly as bayesian analysis would predict. In fact, mathematically it does not make sense for a Bayesian result to contradict a frequentist one, when looking at the same outcomes; if this happens one should suspect that the two problems were formulated using different assumptions, and are thus not equivalent, or that one is looking at different outcomes. I also disagree with the 'mindblowing' example in your blogpost linked above (though I did enjoy reading it, thank you for linking!). Your example only shows the 'superiority' of bayesian reasoning because you (presumably unintentionally) misrepresented the alternative hypothesis by the wrong 'tail' (to keep things simple - one could go further say that there are more effective alternative hypothesis formulations than a 'tailed' one here, since we are not testing simply against the alternative of 'higher bias' but against a known specific bias of 90%). One's choice of alternative hypothesis is part of one's assumptions reflecting prior knowledge. The analysis is misleadingly poor for the frequentist analysis, in that it is equivalent to the bayesian analogue of saying "Assume we have a coin that gives heads 90% of the time and I toss it 10 times; let's represent this with a prior probability Binom(10, 0.1)". What? Of course you can expect such a poor prior to give wrong results with high certainty! It makes no sense to assume that if your null is not true, then you should look for prior knowledge in the lower values. Furthermore, the exact definition of a p value allows great flexibility in the definition of the 'or worse' criterion. You are free to choose a criterion that takes into account prior information (and that forms part of your assumptions). In the 'mindblowing' example, you see that as soon as you choose the ranking measure to be P/(P+Q), and use it to calculate p-values by summing over the probabilities of the 'or worse' cases with regard to that ranking, you get very clear patterns that agree with your bayesian analysis (i.e. that 0 HEADS are more compatible with the unbiased coin hypothesis, than 1 HEADS, and similarly that the reverse applies for the biased hypothesis).
throwaway2245 6y ago
This scenario is several steps too hypothetical for me. If you have 10^1000+1 coins, how did you accurately ascertain that exactly 1 coin was biased and the others were not? And if you have been able to do that, well enough to state it as fact, why do you now need to do this statistical test?
- pontus 6y ago
  It's just a thought experiment meant to highlight a common fallacy in how people think about p-values.
  - deleted 6y ago
    [deleted]
  - throwaway2245 6y ago
    I find it fallacious to see this as a criticism of p-values. If I wanted to identify whether a particular coin was the [most] biased out of 10^1000, flipping one coin 10 times isn't a reasonable setup for the experiment. And, my conclusion to your hypothetical experiment would be: the person who told me that there was a 99.9...% chance this coin was fair was likely incorrect about that.

by pontus 6y ago

nickhuh 6y ago

I think this example misleads one's intuition for the following reason: in the proposed scenario, you'd only see the coin come up heads 10 times in a row about 1 in 1024 times you ran the experiment. While your conclusion would likely be incorrect, you almost never run into that scenario. For example, if you conducted a study every week for 20 years, you'd both be extremely prolific and expect to have drawn about one wrong conclusion. The example is a case of an absurd premise (i.e., a fair coin comes up heads 10 times in a row) leading to an absurd conclusion (that the coin is biased). Of course, this is exactly the guarantee the hypothesis test provides: under robust assumptions, you'll draw the wrong conclusion only rarely.

jbay808 6y ago
A 1 in 1024 chance is not an absurd premise! Something that has a 1 in 1024 chance of ocurring to a person happens to 7 million people. If you get that evidence, you need to be capable of interpreting it correctly. The OP's example would also worked well enough with even just 5 heads; you'd already pass p < 0.05, despite the actual probability of having picked the biased coin still being minuscule.
feanaro 6y ago
I've been playing a board game daily for the last two weeks. As part of this game, a player has to draw one out of six unique cards. In the last five games, I've repeatedly drawn the same card. This is a real example. 1 / (1/6)^5 ≃ 7776
- MaximumYComb 6y ago
  You can't rely on odds after an event has happened to determine probably. I could shuffle a deck containing 1000 unique cards and then look at their order. It's astronomically low that this order occurs but it did happen.
  - joshuamorton 6y ago
    When discussing conditional probability, this is absolutely what you can do. Let's use your example. Conditional probability is P(A|B), the probability of event A, given that event B was observed. What's the probability that I am a magician, given that I shuffled the deck and when you saw it it was still in new deck order? Now certainly, there is an astronomically small chance that this was observed due to random chance. And yet if you observed this, I'd expect that you would, with relatively high confidence, believe that I stacked the deck.
  - feanaro 6y ago
    I'm not doing anything different than what we are doing when discussing the improbability of a hypothetical 10-heads-in-a-row event.
6y ago

tpoacher 6y ago

Except this is not what a p-value is supposed to tell you in the first place. It's supposed to tell you how compatible your observations are with the ASSUMED theory. The minute you use it as an interpretation of a posterior probability (let alone a malformed one), you have violated the assumption on which you rely to interpret it. To use your own example. If you picked a coin from said bag which is mostly full of otherwise unbiased coins, and your hypothesis is "this coin is not biased", a sequence of ten heads would be largely "incompatible" with your hypothesis (as if the hypothesis of unbiasedness was true, you should only expect this result 1/1024 times). It is up to you to then place this result in the context of other prior assumptions etc, and say "yes it's incompatible but still very likely". The p-value does not preclude you from performing a post-hoc analysis of the base-rate if you so wish. It just so happens that the 'assume a hypothesis' approach is a useful one in the absence of all other information. Which is not so different from a Bayesian saying 'I'll use an uninformative prior and let the data speak for itself'. But using it as a test of posterior probability, especially when prior information exists, and saying "ha see it sucks" is not a fair criticism for it (whereas criticising its use as a test of posterior probability is).

pontus 6y ago
Yes, that's exactly my point. Saying that it really supposed to be the probability of the data assuming the null hypothesis is true is pretty confusing to most people and often times the gut feeling people keep us that it still somehow tells you how likely it is that the null hypothesis is true. I just like this example because it highlights that there is a distinction between these two probabilities.
pontus 6y ago
In fact, if you look at the document I link to above, you'll see the correct Bayesian calculation where we actually compute p(Biased | 10H). In the first case it's pretty high but in the second case it's very small.
- tpoacher 6y ago
  I 100% agree that p-values are hard to get right (not that Bayesian analysis is necessarily more straightforward, mind you). What I disagree with is the assertion that a) the two scenarios listed above are equivalent, and b) that you cannot incorporate a prior analysis for use with p-values (which is best thought simply as a measures of compatibility of the data with respect to a hypothesis). E.g. in the biased vs unbiased example, if your null hypothesis is "The coin is biased", then it is simply the wrong hypothesis for the problem (or, more specifically, the wrong null/alternative pair, since these two only make sense as a pair; a null by itself is not sufficient to define the assumptions for the problem). If your alternative hypothesis is "ok, it's one of the 10^1000-1 unbiased ones", then your null should have necessarily been "it's the biased coin with 1/10^1000 chance of being chosen", which effectively forces you to apply your prior probabilities on the hypotheses themselves. Once you do that, you will easily see that the compatibility of the data with the alternative hypothesis is much higher, exactly as bayesian analysis would predict. In fact, mathematically it does not make sense for a Bayesian result to contradict a frequentist one, when looking at the same outcomes; if this happens one should suspect that the two problems were formulated using different assumptions, and are thus not equivalent, or that one is looking at different outcomes. I also disagree with the 'mindblowing' example in your blogpost linked above (though I did enjoy reading it, thank you for linking!). Your example only shows the 'superiority' of bayesian reasoning because you (presumably unintentionally) misrepresented the alternative hypothesis by the wrong 'tail' (to keep things simple - one could go further say that there are more effective alternative hypothesis formulations than a 'tailed' one here, since we are not testing simply against the alternative of 'higher bias' but against a known specific bias of 90%). One's choice of alternative hypothesis is part of one's assumptions reflecting prior knowledge. The analysis is misleadingly poor for the frequentist analysis, in that it is equivalent to the bayesian analogue of saying "Assume we have a coin that gives heads 90% of the time and I toss it 10 times; let's represent this with a prior probability Binom(10, 0.1)". What? Of course you can expect such a poor prior to give wrong results with high certainty! It makes no sense to assume that if your null is not true, then you should look for prior knowledge in the lower values. Furthermore, the exact definition of a p value allows great flexibility in the definition of the 'or worse' criterion. You are free to choose a criterion that takes into account prior information (and that forms part of your assumptions). In the 'mindblowing' example, you see that as soon as you choose the ranking measure to be P/(P+Q), and use it to calculate p-values by summing over the probabilities of the 'or worse' cases with regard to that ranking, you get very clear patterns that agree with your bayesian analysis (i.e. that 0 HEADS are more compatible with the unbiased coin hypothesis, than 1 HEADS, and similarly that the reverse applies for the biased hypothesis).

throwaway2245 6y ago

This scenario is several steps too hypothetical for me. If you have 10^1000+1 coins, how did you accurately ascertain that exactly 1 coin was biased and the others were not? And if you have been able to do that, well enough to state it as fact, why do you now need to do this statistical test?

pontus 6y ago
It's just a thought experiment meant to highlight a common fallacy in how people think about p-values.
- deleted 6y ago
  [deleted]
- throwaway2245 6y ago
  I find it fallacious to see this as a criticism of p-values. If I wanted to identify whether a particular coin was the [most] biased out of 10^1000, flipping one coin 10 times isn't a reasonable setup for the experiment. And, my conclusion to your hypothetical experiment would be: the person who told me that there was a 99.9...% chance this coin was fair was likely incorrect about that.