5 ms·

The language of statistics is indeed dishonest. In normal language "significant" has a meaning close to "large". What they mean in statistics is that there is a

The language of statistics is indeed dishonest. In normal language "significant" has a meaning close to "large". What they mean in statistics is that there is a detectable signal, there is a 95% chance that the data is not pure noise. They should call it "detectable", some people have suggested "discernible". However, there is another fatal flaw to how p-values are used. They are usually used for rejecting infinitesimally small hypothesis. The null hypothesis is stated as "Effect exactly equals 0.00000000000...". In practice, there are no experiments that have exactly zero effect. There are always at least very small systematic bias due to imperfectly calibrated instruments or small methodological variations between researchers. Even if you do everything else right, if you pre-register your study, with enough data, a null hypothesis test will always pick up on these small biases and make the results significant. If you are looking to reject a null hypothesis, I can tell you in advance: all experiments have a non zero bias, all results are statistically significant p<0.000000001 with enough data. There, I just saved the scientific world a ton of money, they don't have to do all these experiments. Just reference this comment in your paper to show significance. At least here, the language seems honest. Rejecting a null hypothesis correctly conveys the idea of having rejected nothing. Using a Popperian approach is great, but you should reject a portion of the hypothesis space this is bigger than zero.

maxnoe 8y ago
"They" is a pretty broad term. Physicist's convention is to call a 99.7 (3 sigma) per cent chance a "hint" and only a 5 sigma effect a detection.
joshuamorton 8y ago
This isn't the case. I can make two hypotheses that are contradictory. Both aren't going to be statistically significant from the same data, no matter how biased my equipment or the size of the sample.
- BenoitEssiambre 8y ago
  Yes, the problem is that with null hypothesis testing, the two contradictory hypothesis are usually: "effect is exactly 0.00000000..." and "there is an effect different than 0.000000000...". The problem stems from the fact that people pit an infinitesimally small hypothesis against an infinitely large one. Rejecting an infinitesimally small hypothesis is very very easy. The tiniest of experimental bias will allow rejection if you have enough data. It would be impressive if "There is an effect different than zero" was rejected (but no one would ever be able to do this). Science should try to reject finitely large hypotheses, at a minimum something like: "effect is larger than some reasonable margin to account for experimental bias and imperfect tools". At least a small chunk of the hypothesis space should be rejected for your experiment to be worth something. You sorta get that with confidence intervals since you can see how far the lower bound is from zero.
  - joshuamorton 8y ago
    I mean, you can still run I to issues here: The effect size is <1% and the effect size is >=1%. Or 100 different hypitheses for 1-100% effect sizes. They're mutually exclusive, so only one will be true. So again, while in general I agree with you that beyesian methods have significant advantages, this objection isn't well founded.

maxnoe 8y ago

"They" is a pretty broad term. Physicist's convention is to call a 99.7 (3 sigma) per cent chance a "hint" and only a 5 sigma effect a detection.

joshuamorton 8y ago

This isn't the case. I can make two hypotheses that are contradictory. Both aren't going to be statistically significant from the same data, no matter how biased my equipment or the size of the sample.

BenoitEssiambre 8y ago
Yes, the problem is that with null hypothesis testing, the two contradictory hypothesis are usually: "effect is exactly 0.00000000..." and "there is an effect different than 0.000000000...". The problem stems from the fact that people pit an infinitesimally small hypothesis against an infinitely large one. Rejecting an infinitesimally small hypothesis is very very easy. The tiniest of experimental bias will allow rejection if you have enough data. It would be impressive if "There is an effect different than zero" was rejected (but no one would ever be able to do this). Science should try to reject finitely large hypotheses, at a minimum something like: "effect is larger than some reasonable margin to account for experimental bias and imperfect tools". At least a small chunk of the hypothesis space should be rejected for your experiment to be worth something. You sorta get that with confidence intervals since you can see how far the lower bound is from zero.
- joshuamorton 8y ago
  I mean, you can still run I to issues here: The effect size is <1% and the effect size is >=1%. Or 100 different hypitheses for 1-100% effect sizes. They're mutually exclusive, so only one will be true. So again, while in general I agree with you that beyesian methods have significant advantages, this objection isn't well founded.