4 ms·
This test seems really underrated. It's my go-to for comparing computer system performance (e.g. between versions on CI) since they often have very peculiar dis
by lukego 4y ago
This test seems really underrated. It's my go-to for comparing computer system performance (e.g. between versions on CI) since they often have very peculiar distributions and are relatively cheap to produce enough samples from.
- mapme 4y agoHmm, I was trying to do the same with KS test on performance data but it seemed extremely sensitive to outliers even when an eyeball test of the two distributions look near exact. Have you ran into any of those issues?
- lukego 4y agoNot with outliers, no, I think it handles them particularly well (conservatively.) But yes to small consistent differences (e.g. uniform 1%) that affect the relative order of results but not by an very important amount. So have to consider effect size even when the test statistic is strong.
- bigbillheck 4y agoI've stopped using it as I found it far too sensitive to small differences.
- klipt 4y agoPerhaps this comes from a misunderstanding of what statistical significance means. A test reporting a statistically significant difference doesn't mean the difference is big, just that it's big enough to separate out as a "signal" from the underlying random noise. It's basically saying "yes, given this data I am very confident that there is an underlying difference that is not just an artefact of random sampling".
- lanstin 4y agoWhich if you have a million samples can be true without the statement “these are basically acting the same” being falsified in some human sense.
- lukego 4y agoYeah true dat. Still, if you have 5-50 samples from an unknown distribution I think the KS test can be a very valuable protection against false-positive i.e. seeing patterns in random noise. Even more so if you are making multiple comparisons, e.g. comparing multiple response variables, since you can easily make a Bonferroni correction to avoid "the xkcd 882 problem" which is hard to do by eyeball. So I see a lot of utility in the KS test (and Wilcox, etc) when measuring computer systems on multiple metrics with unknown distributions (often not Gaussian) and where samples cost ~$0.10-$1.0 each. I see a lot of false positives in the absence of such tests, or otherwise generally low confidence in inferences based on e.g. eyeball/mean/median.
- bigbillheck 4y agoThis is the kind of thing I brought up in a sibling comment about 'why do you care?' Do you care about the distribution of some metric, or do you care about location or scale?
- antman 4y agoWhat do you use instead?
- bigbillheck 4y agoIf I care about 'was this data generated by this specific distribution' then I'll quantize and do a chi-square. If I care about 'were these two datasets generated by the same distribution' then I'll step back and ask 'why do I care?' and 'what will I do with the answer?' and then use a more specific test.