4 ms·
Perhaps this comes from a misunderstanding of what statistical significance means. A test reporting a statistically significant difference doesn't mean the diff
by klipt 4y ago
Perhaps this comes from a misunderstanding of what statistical significance means. A test reporting a statistically significant difference doesn't mean the difference is big, just that it's big enough to separate out as a "signal" from the underlying random noise.
It's basically saying "yes, given this data I am very confident that there is an underlying difference that is not just an artefact of random sampling".
- lanstin 4y agoWhich if you have a million samples can be true without the statement “these are basically acting the same” being falsified in some human sense.
- lukego 4y agoYeah true dat. Still, if you have 5-50 samples from an unknown distribution I think the KS test can be a very valuable protection against false-positive i.e. seeing patterns in random noise. Even more so if you are making multiple comparisons, e.g. comparing multiple response variables, since you can easily make a Bonferroni correction to avoid "the xkcd 882 problem" which is hard to do by eyeball. So I see a lot of utility in the KS test (and Wilcox, etc) when measuring computer systems on multiple metrics with unknown distributions (often not Gaussian) and where samples cost ~$0.10-$1.0 each. I see a lot of false positives in the absence of such tests, or otherwise generally low confidence in inferences based on e.g. eyeball/mean/median.
- bigbillheck 4y agoThis is the kind of thing I brought up in a sibling comment about 'why do you care?' Do you care about the distribution of some metric, or do you care about location or scale?