3 ms·

> I suppose 'chance' is a little hand-wavy, but isn't a p-value just the probability of your data given that your hypothesis is false? It is the probability of

> I suppose 'chance' is a little hand-wavy, but isn't a p-value just the probability of your data given that your hypothesis is false? It is the probability of achieving a result at least as far from the hypothesized value exclusively due to random variation that is uncorrelated with the explanatory variable(s) at hand [and subject to a number of other assumptions].

darawk 11y ago
Ya, but is that not equivalent to the statement 'due to chance'? I think the common understanding of 'chance' would include 'due to random variation uncorrelated with the explanatory variables'.
- chimeracoder 11y ago
  > I think the common understanding of 'chance' would include 'due to random variation uncorrelated with the explanatory variables'. Yes, but the problem is that it also includes more than that. One of the many problems with p-values is that people assume that the p-value encodes a lot more information than it actually does. Another problem: it doesn't really tell you any "new" information. All equality-based null hypotheses are false, and we know this due to continuity theory[0]. Really, all a hypothesis test does is tell us if the sample size is large enough to reflect this knowledge. Literally any null hypothesis can be rejected, as long as the sample size is sufficiently large. It's worse than that, though. Yes, hypothesis testing doesn't tell us anything about the practical significance of the results. But it also doesn't actually tell us what most people thing it does: a measure of how wrong our hypothesis is. Rejection is a binary state: there's no concept of "strongly rejecting" a null hypothesis[1]. [0] We could reject continuity on the basis that the real world is actually discrete at the quantum level, but a lot of the math underpinning hypothesis testing falls apart if you don't assume continuity, so it's broken either way. [1] Many people - including statisticians - will sometimes imply this colloquially ("the p-value was .00001, so our hypothesis was completely wrong"). It's not always "wrong", because largely you can measure this concept in other ways, but you cannot do so with a p-value.

darawk 11y ago

Ya, but is that not equivalent to the statement 'due to chance'? I think the common understanding of 'chance' would include 'due to random variation uncorrelated with the explanatory variables'.

chimeracoder 11y ago
> I think the common understanding of 'chance' would include 'due to random variation uncorrelated with the explanatory variables'. Yes, but the problem is that it also includes more than that. One of the many problems with p-values is that people assume that the p-value encodes a lot more information than it actually does. Another problem: it doesn't really tell you any "new" information. All equality-based null hypotheses are false, and we know this due to continuity theory[0]. Really, all a hypothesis test does is tell us if the sample size is large enough to reflect this knowledge. Literally any null hypothesis can be rejected, as long as the sample size is sufficiently large. It's worse than that, though. Yes, hypothesis testing doesn't tell us anything about the practical significance of the results. But it also doesn't actually tell us what most people thing it does: a measure of how wrong our hypothesis is. Rejection is a binary state: there's no concept of "strongly rejecting" a null hypothesis[1]. [0] We could reject continuity on the basis that the real world is actually discrete at the quantum level, but a lot of the math underpinning hypothesis testing falls apart if you don't assume continuity, so it's broken either way. [1] Many people - including statisticians - will sometimes imply this colloquially ("the p-value was .00001, so our hypothesis was completely wrong"). It's not always "wrong", because largely you can measure this concept in other ways, but you cannot do so with a p-value.