3 ms·

Except this is not what a p-value is supposed to tell you in the first place. It's supposed to tell you how compatible your observations are with the ASSUMED th

Except this is not what a p-value is supposed to tell you in the first place. It's supposed to tell you how compatible your observations are with the ASSUMED theory. The minute you use it as an interpretation of a posterior probability (let alone a malformed one), you have violated the assumption on which you rely to interpret it. To use your own example. If you picked a coin from said bag which is mostly full of otherwise unbiased coins, and your hypothesis is "this coin is not biased", a sequence of ten heads would be largely "incompatible" with your hypothesis (as if the hypothesis of unbiasedness was true, you should only expect this result 1/1024 times). It is up to you to then place this result in the context of other prior assumptions etc, and say "yes it's incompatible but still very likely". The p-value does not preclude you from performing a post-hoc analysis of the base-rate if you so wish. It just so happens that the 'assume a hypothesis' approach is a useful one in the absence of all other information. Which is not so different from a Bayesian saying 'I'll use an uninformative prior and let the data speak for itself'. But using it as a test of posterior probability, especially when prior information exists, and saying "ha see it sucks" is not a fair criticism for it (whereas criticising its use as a test of posterior probability is).

pontus 6y ago
Yes, that's exactly my point. Saying that it really supposed to be the probability of the data assuming the null hypothesis is true is pretty confusing to most people and often times the gut feeling people keep us that it still somehow tells you how likely it is that the null hypothesis is true. I just like this example because it highlights that there is a distinction between these two probabilities.
pontus 6y ago
In fact, if you look at the document I link to above, you'll see the correct Bayesian calculation where we actually compute p(Biased | 10H). In the first case it's pretty high but in the second case it's very small.
- tpoacher 6y ago
  I 100% agree that p-values are hard to get right (not that Bayesian analysis is necessarily more straightforward, mind you). What I disagree with is the assertion that a) the two scenarios listed above are equivalent, and b) that you cannot incorporate a prior analysis for use with p-values (which is best thought simply as a measures of compatibility of the data with respect to a hypothesis). E.g. in the biased vs unbiased example, if your null hypothesis is "The coin is biased", then it is simply the wrong hypothesis for the problem (or, more specifically, the wrong null/alternative pair, since these two only make sense as a pair; a null by itself is not sufficient to define the assumptions for the problem). If your alternative hypothesis is "ok, it's one of the 10^1000-1 unbiased ones", then your null should have necessarily been "it's the biased coin with 1/10^1000 chance of being chosen", which effectively forces you to apply your prior probabilities on the hypotheses themselves. Once you do that, you will easily see that the compatibility of the data with the alternative hypothesis is much higher, exactly as bayesian analysis would predict. In fact, mathematically it does not make sense for a Bayesian result to contradict a frequentist one, when looking at the same outcomes; if this happens one should suspect that the two problems were formulated using different assumptions, and are thus not equivalent, or that one is looking at different outcomes. I also disagree with the 'mindblowing' example in your blogpost linked above (though I did enjoy reading it, thank you for linking!). Your example only shows the 'superiority' of bayesian reasoning because you (presumably unintentionally) misrepresented the alternative hypothesis by the wrong 'tail' (to keep things simple - one could go further say that there are more effective alternative hypothesis formulations than a 'tailed' one here, since we are not testing simply against the alternative of 'higher bias' but against a known specific bias of 90%). One's choice of alternative hypothesis is part of one's assumptions reflecting prior knowledge. The analysis is misleadingly poor for the frequentist analysis, in that it is equivalent to the bayesian analogue of saying "Assume we have a coin that gives heads 90% of the time and I toss it 10 times; let's represent this with a prior probability Binom(10, 0.1)". What? Of course you can expect such a poor prior to give wrong results with high certainty! It makes no sense to assume that if your null is not true, then you should look for prior knowledge in the lower values. Furthermore, the exact definition of a p value allows great flexibility in the definition of the 'or worse' criterion. You are free to choose a criterion that takes into account prior information (and that forms part of your assumptions). In the 'mindblowing' example, you see that as soon as you choose the ranking measure to be P/(P+Q), and use it to calculate p-values by summing over the probabilities of the 'or worse' cases with regard to that ranking, you get very clear patterns that agree with your bayesian analysis (i.e. that 0 HEADS are more compatible with the unbiased coin hypothesis, than 1 HEADS, and similarly that the reverse applies for the biased hypothesis).

pontus 6y ago

Yes, that's exactly my point. Saying that it really supposed to be the probability of the data assuming the null hypothesis is true is pretty confusing to most people and often times the gut feeling people keep us that it still somehow tells you how likely it is that the null hypothesis is true. I just like this example because it highlights that there is a distinction between these two probabilities.

pontus 6y ago

In fact, if you look at the document I link to above, you'll see the correct Bayesian calculation where we actually compute p(Biased | 10H). In the first case it's pretty high but in the second case it's very small.

tpoacher 6y ago
I 100% agree that p-values are hard to get right (not that Bayesian analysis is necessarily more straightforward, mind you). What I disagree with is the assertion that a) the two scenarios listed above are equivalent, and b) that you cannot incorporate a prior analysis for use with p-values (which is best thought simply as a measures of compatibility of the data with respect to a hypothesis). E.g. in the biased vs unbiased example, if your null hypothesis is "The coin is biased", then it is simply the wrong hypothesis for the problem (or, more specifically, the wrong null/alternative pair, since these two only make sense as a pair; a null by itself is not sufficient to define the assumptions for the problem). If your alternative hypothesis is "ok, it's one of the 10^1000-1 unbiased ones", then your null should have necessarily been "it's the biased coin with 1/10^1000 chance of being chosen", which effectively forces you to apply your prior probabilities on the hypotheses themselves. Once you do that, you will easily see that the compatibility of the data with the alternative hypothesis is much higher, exactly as bayesian analysis would predict. In fact, mathematically it does not make sense for a Bayesian result to contradict a frequentist one, when looking at the same outcomes; if this happens one should suspect that the two problems were formulated using different assumptions, and are thus not equivalent, or that one is looking at different outcomes. I also disagree with the 'mindblowing' example in your blogpost linked above (though I did enjoy reading it, thank you for linking!). Your example only shows the 'superiority' of bayesian reasoning because you (presumably unintentionally) misrepresented the alternative hypothesis by the wrong 'tail' (to keep things simple - one could go further say that there are more effective alternative hypothesis formulations than a 'tailed' one here, since we are not testing simply against the alternative of 'higher bias' but against a known specific bias of 90%). One's choice of alternative hypothesis is part of one's assumptions reflecting prior knowledge. The analysis is misleadingly poor for the frequentist analysis, in that it is equivalent to the bayesian analogue of saying "Assume we have a coin that gives heads 90% of the time and I toss it 10 times; let's represent this with a prior probability Binom(10, 0.1)". What? Of course you can expect such a poor prior to give wrong results with high certainty! It makes no sense to assume that if your null is not true, then you should look for prior knowledge in the lower values. Furthermore, the exact definition of a p value allows great flexibility in the definition of the 'or worse' criterion. You are free to choose a criterion that takes into account prior information (and that forms part of your assumptions). In the 'mindblowing' example, you see that as soon as you choose the ranking measure to be P/(P+Q), and use it to calculate p-values by summing over the probabilities of the 'or worse' cases with regard to that ranking, you get very clear patterns that agree with your bayesian analysis (i.e. that 0 HEADS are more compatible with the unbiased coin hypothesis, than 1 HEADS, and similarly that the reverse applies for the biased hypothesis).