4 ms·

a p-value of 0.05 or even 0.01 is stupidly high. it only takes a little thought experiment about what that means in reality to realise how permissive it is and

by jheriko 13y ago

a p-value of 0.05 or even 0.01 is stupidly high. it only takes a little thought experiment about what that means in reality to realise how permissive it is and you can find demonstrations of this without going particularly far, looking very hard or being especially well educated... consider the wikipedia example with heads vs. tails. http://en.wikipedia.org/wiki/P-value#Examples http://en.wikipedia.org/wiki/P-value#Examples the idea that 5 coin tosses can produce a p-value < 0.05 that 'demonstrates' that the coin is biased towards heads is intuitively 'obviously wrong'. even if we take it to 10 coin tosses (the p-value you get is 0.001 - which looks really strong if we accept that 0.01 is acceptable) it clashes with my own ideals for what statistical significance should mean. this is in a loose way a proof by contradiction that p-values of 0.05 or 0.01 do not have utility (at least for these kinds of small n). aside from that consider running the experiment 5 times or 20 times. how many false positives do you expect? what is the expected number of false positives? is that significant? it also bothers me how connected to the problem formulation that the value itself is. if we analyse the same situation with an identical test but a different formulation of the problem that the values differ? why is five heads in a row less significant as a result when the test is whether a coin is biased at all rather than a test that it is biased towards heads only? sure i understand the probability involved there that we have all these potential coins biased towards tails that mean nothing in the first case - but there is something very deeply wrong with that. shouldn't this be the other way around? if 5 consecutive heads is good evidence that a coin is biased towards heads, isn't it equally good evidence that it is biased at all? classical logic says that it is because being biased towards heads is a subset of being biased in either direction. the truth is that it really is equally good evidence - i challenge someone to explain why it is not! ( actually i kinda want to be wrong about that because i might learn something new then :) ) probability is counter-intuitive and useless for the kinds of small n usually used in experiments - the intuition about it recovers when we deal with sensible n - numbers like 1000 or 10000 - but these are still small n really if you need to scale up, or be confident that your result is correct. even at 100 samples its obvious that our idealisation of percentage and what happens in reality do not marry up neatly... to make a very crude software analogy what about those 1 in 10,000 bugs? they are still a very real problem if you have millions of customers... or - IMO even 10,000 is a very exceedingly small n to try and draw robust conclusions from.

sp332 13y ago
0.05 == 5% == 1/20. If you flip a coin 5 times and get heads every time, do you intuitively feel that there is more than 1-in-20 odds that the coin is fair? You should really get used to the idea that stating a different problem will give you a different answer. You need to be very careful when asking a question, or your answer might not mean what you think it means.
- Sharlin 13y ago
  Bayes to the rescue! In this case, most people's prior, that is, the unconditional probability that a given coin is biased, is probably much less than 0.5 - after all, almost all coins are supposed to be nonbiased, and biased coins must be specially manufactured. In light of this prior, after five heads in a row, the posterior probability that a randomly chosen coin is biased is much less than 0.95. On the other hand if your prior really is 0.5, for instance if you have two coins and know one is biased but not which one, then, after five heads, you really should deduce that the probability of you having picked the biased one is high (it depends on how biased exactly the coin is!)
- jheriko 13y ago
  maybe my presentation was unclear. i seem to have given the impression that i am really quite dim... > You should really get used to the idea that stating a different problem will give you a different answer this is entirely normal and expected since ever i can remember... what i'm saying is that you can analyse the same data in two different ways and reach differing conclusions because of the nature of the p-value (vs. the nature of, well... nature) what i was mainly trying to get across is that a coin being biased towards heads /logically infers that/ it is biased. so the idea that 5 heads in a row is less evidence of a coin being biased than it is that it is biased towards heads is not only counter-intuitive but in disagreement with a much stronger and more intuitive form of reasoning. the fact that the p-values are different in these cases leads me to expect that p-values on their own are not a good indicator of strength of evidence without a lot more context - and /really/ understanding what that context is and means - in which case why use the value at all? nobody else is likely to interpret it correctly unless you lay it out that way which then negates the supposed utility of the p-value... and yes, i intuitively consider 5 heads in a row to be unspectacular for a fair coin certainly not a 19/20 chance that it is biased (maybe i am very, very wrong though).

by jheriko 13y ago

sp332 13y ago

0.05 == 5% == 1/20. If you flip a coin 5 times and get heads every time, do you intuitively feel that there is more than 1-in-20 odds that the coin is fair? You should really get used to the idea that stating a different problem will give you a different answer. You need to be very careful when asking a question, or your answer might not mean what you think it means.

Sharlin 13y ago
Bayes to the rescue! In this case, most people's prior, that is, the unconditional probability that a given coin is biased, is probably much less than 0.5 - after all, almost all coins are supposed to be nonbiased, and biased coins must be specially manufactured. In light of this prior, after five heads in a row, the posterior probability that a randomly chosen coin is biased is much less than 0.95. On the other hand if your prior really is 0.5, for instance if you have two coins and know one is biased but not which one, then, after five heads, you really should deduce that the probability of you having picked the biased one is high (it depends on how biased exactly the coin is!)
jheriko 13y ago
maybe my presentation was unclear. i seem to have given the impression that i am really quite dim... > You should really get used to the idea that stating a different problem will give you a different answer this is entirely normal and expected since ever i can remember... what i'm saying is that you can analyse the same data in two different ways and reach differing conclusions because of the nature of the p-value (vs. the nature of, well... nature) what i was mainly trying to get across is that a coin being biased towards heads /logically infers that/ it is biased. so the idea that 5 heads in a row is less evidence of a coin being biased than it is that it is biased towards heads is not only counter-intuitive but in disagreement with a much stronger and more intuitive form of reasoning. the fact that the p-values are different in these cases leads me to expect that p-values on their own are not a good indicator of strength of evidence without a lot more context - and /really/ understanding what that context is and means - in which case why use the value at all? nobody else is likely to interpret it correctly unless you lay it out that way which then negates the supposed utility of the p-value... and yes, i intuitively consider 5 heads in a row to be unspectacular for a fair coin certainly not a 19/20 chance that it is biased (maybe i am very, very wrong though).