3 ms·
My biggest pet peeve is the interpretation of p-values absent the context of effect sizes. If you have a huge sample size you're quite frequently going to find
by glaugh 13y ago
My biggest pet peeve is the interpretation of p-values absent the context of effect sizes. If you have a huge sample size you're quite frequently going to find statistically significant differences between groups, but often those differences aren't that meaningful.
If you assessed differences in average human height between two cities based on a million datapoints, you're pretty darn likely to find a difference that's statistically significant but not really important or meaningful (like a .1mm difference in average height).
The above is why the American Psychological Association says "reporting and interpreting effect sizes in the context of [p-values] is essential to good research".[1] It's also why Statwing always report effect sizes along with p-values for hypothesis tests.[2]
(For clarity: effect size is best presented in readily interpretable, concrete terms like height or whatever unit you're using. If that's not possible because you're comparing ratings on a 1 to 7 scale or something, or if you want to compare effect sizes across different types of analyses, there are specific metrics of effect size).
[1] http://people.cehd.tamu.edu/~bthompson/apaeffec.htm http://people.cehd.tamu.edu/~bthompson/apaeffec.htm
[2] https://www.statwing.com/demo https://www.statwing.com/demo
- btilly 13y agoThe effect sizes that you're talking about are the result of hidden correlations causing the random variation to be larger than the statistical model thought possible. On the one hand you can always argue that this is a sign you did statistics wrong. On the other hand the internal correlations caused by ethnicity, diet, environmental pollution, etc truly are hard to estimate. Another common cause of finding significant p-values when you shouldn't is looking at many metrics (or the same metric at many points of time). Eventually you get (un)lucky. However if you do the stats correctly, and the statistical model (which usually assumes independence) is accurate, then a p-value of under 5% will only happen 5% of the time. No matter how big the sample size is.
- glaugh 13y agoYeah, my point isn't that the statistical significance is erroneous or that the difference shouldn't have been significant. In my totally made up example, there's really a .1mm difference. And maybe to someone that's meaningful, so it's worth reporting. But it's not a big enough effect to say "When I see someone on the street I can look at their height and know which of these two cities they're from." And my peeve is that people will see a statistically significant difference and think that that alone makes it an important finding, versus a very real finding that might not actually matter that much. Or, related, it's bad to see something with a p-value of .15 and a large effect size and decide that since it's not significant at .05 it's not interesting. Since that combination typically means that you have a too-small sample size, the best interpretation is probably more like "This might be interesting. Looks like we don't have enough data to tell if there's a relationship here, but it probably deserves another look." (Edited to add that last thought, and clarify the previous one)
- capnrefsmmat 13y agoThe other big problem is studies which do report effect sizes but do not report their statistical power. If your study is too small to detect the true effect, the only possible way you can achieve significance is by getting lucky and detecting an exaggerated effect -- and in many fields the average study is far too small to detect the true effects. I'm halfway through a major revision of my guide to statistical error that discusses this topic: http://www.refsmmat.com/statistics/regression.html#truth-inflation http://www.refsmmat.com/statistics/regression.html#truth-inf...