4 ms·
One of the joys of programming is that, unlike maths, when discovering an exciting new paradigm there is an opportunity to rationalise all the notation and use
by roenxi 8y ago
One of the joys of programming is that, unlike maths, when discovering an exciting new paradigm there is an opportunity to rationalise all the notation and use a new language.
Contrasting, I struggle to deal with probability and statistics without developing a strong suspicion that the name the objects are called by is completely different from what the name means in common English.
It is nice to see ongoing authoritative commentary that the large majority do not understand what a p-value actually implies. The thread of discourse seems to be that even assuming that all the academics are completely honest (ie, no academic fraud, no hand waving) the number of false results that are awarded statistical significance is much higher than it should be. The standard p value threshold at 5% does not imply that 95% of the statistically significant studies are not by chance. Particularly amongst the subset that make it into the public eye.
- BenoitEssiambre 8y agoThe language of statistics is indeed dishonest. In normal language "significant" has a meaning close to "large". What they mean in statistics is that there is a detectable signal, there is a 95% chance that the data is not pure noise. They should call it "detectable", some people have suggested "discernible". However, there is another fatal flaw to how p-values are used. They are usually used for rejecting infinitesimally small hypothesis. The null hypothesis is stated as "Effect exactly equals 0.00000000000...". In practice, there are no experiments that have exactly zero effect. There are always at least very small systematic bias due to imperfectly calibrated instruments or small methodological variations between researchers. Even if you do everything else right, if you pre-register your study, with enough data, a null hypothesis test will always pick up on these small biases and make the results significant. If you are looking to reject a null hypothesis, I can tell you in advance: all experiments have a non zero bias, all results are statistically significant p<0.000000001 with enough data. There, I just saved the scientific world a ton of money, they don't have to do all these experiments. Just reference this comment in your paper to show significance. At least here, the language seems honest. Rejecting a null hypothesis correctly conveys the idea of having rejected nothing. Using a Popperian approach is great, but you should reject a portion of the hypothesis space this is bigger than zero.
- maxnoe 8y ago"They" is a pretty broad term. Physicist's convention is to call a 99.7 (3 sigma) per cent chance a "hint" and only a 5 sigma effect a detection.
- joshuamorton 8y agoThis isn't the case. I can make two hypotheses that are contradictory. Both aren't going to be statistically significant from the same data, no matter how biased my equipment or the size of the sample.
- BenoitEssiambre 8y agoYes, the problem is that with null hypothesis testing, the two contradictory hypothesis are usually: "effect is exactly 0.00000000..." and "there is an effect different than 0.000000000...". The problem stems from the fact that people pit an infinitesimally small hypothesis against an infinitely large one. Rejecting an infinitesimally small hypothesis is very very easy. The tiniest of experimental bias will allow rejection if you have enough data. It would be impressive if "There is an effect different than zero" was rejected (but no one would ever be able to do this). Science should try to reject finitely large hypotheses, at a minimum something like: "effect is larger than some reasonable margin to account for experimental bias and imperfect tools". At least a small chunk of the hypothesis space should be rejected for your experiment to be worth something. You sorta get that with confidence intervals since you can see how far the lower bound is from zero.
- joshuamorton 8y agoI mean, you can still run I to issues here: The effect size is <1% and the effect size is >=1%. Or 100 different hypitheses for 1-100% effect sizes. They're mutually exclusive, so only one will be true. So again, while in general I agree with you that beyesian methods have significant advantages, this objection isn't well founded.
- FabHK 8y agoNot sure I follow you there. Are you suggesting that better terminology would help? Even if there were better names, it's a pretty hard to change words that are so entrenched. At any rate, "p-value" is a made up, artificial word, certainly better than using a common existing word (such as "likelihood" or "significance") which would be even easier to misunderstand. The fundamental problem is that hypothesis testing happens within a whole theoretical framework, and the jargon refers to things well defined and understood within that framework. I think there is just no way of breaking it down further (though the implications and typical limitations of the research could maybe be communicated better).
- diffeomorphism 8y ago> One of the joys of programming is that, unlike maths, This does happen in maths all the time. Pretty much all new results and new techniques also introduce new notation and language, which is then rationalized in further works by the community. What tends to not change all that much currently is notation used in undergrad textbooks. However, going back more than a few decades books on for instance calculus look very different and use very different language (e.g. infinitesimals, topology, focus on power series etc.), similarly for linear algebra or complex analysis.