4 ms·

I've stopped using it as I found it far too sensitive to small differences.

klipt 4y ago
Perhaps this comes from a misunderstanding of what statistical significance means. A test reporting a statistically significant difference doesn't mean the difference is big, just that it's big enough to separate out as a "signal" from the underlying random noise. It's basically saying "yes, given this data I am very confident that there is an underlying difference that is not just an artefact of random sampling".
- lanstin 4y ago
  Which if you have a million samples can be true without the statement “these are basically acting the same” being falsified in some human sense.
  - lukego 4y ago
    Yeah true dat. Still, if you have 5-50 samples from an unknown distribution I think the KS test can be a very valuable protection against false-positive i.e. seeing patterns in random noise. Even more so if you are making multiple comparisons, e.g. comparing multiple response variables, since you can easily make a Bonferroni correction to avoid "the xkcd 882 problem" which is hard to do by eyeball. So I see a lot of utility in the KS test (and Wilcox, etc) when measuring computer systems on multiple metrics with unknown distributions (often not Gaussian) and where samples cost ~$0.10-$1.0 each. I see a lot of false positives in the absence of such tests, or otherwise generally low confidence in inferences based on e.g. eyeball/mean/median.
    - bigbillheck 4y ago
      This is the kind of thing I brought up in a sibling comment about 'why do you care?' Do you care about the distribution of some metric, or do you care about location or scale?
antman 4y ago
What do you use instead?
- bigbillheck 4y ago
  If I care about 'was this data generated by this specific distribution' then I'll quantize and do a chi-square. If I care about 'were these two datasets generated by the same distribution' then I'll step back and ask 'why do I care?' and 'what will I do with the answer?' and then use a more specific test.

Perhaps this comes from a misunderstanding of what statistical significance means. A test reporting a statistically significant difference doesn't mean the difference is big, just that it's big enough to separate out as a "signal" from the underlying random noise. It's basically saying "yes, given this data I am very confident that there is an underlying difference that is not just an artefact of random sampling".

lanstin 4y ago
Which if you have a million samples can be true without the statement “these are basically acting the same” being falsified in some human sense.
- lukego 4y ago
  Yeah true dat. Still, if you have 5-50 samples from an unknown distribution I think the KS test can be a very valuable protection against false-positive i.e. seeing patterns in random noise. Even more so if you are making multiple comparisons, e.g. comparing multiple response variables, since you can easily make a Bonferroni correction to avoid "the xkcd 882 problem" which is hard to do by eyeball. So I see a lot of utility in the KS test (and Wilcox, etc) when measuring computer systems on multiple metrics with unknown distributions (often not Gaussian) and where samples cost ~$0.10-$1.0 each. I see a lot of false positives in the absence of such tests, or otherwise generally low confidence in inferences based on e.g. eyeball/mean/median.
  - bigbillheck 4y ago
    This is the kind of thing I brought up in a sibling comment about 'why do you care?' Do you care about the distribution of some metric, or do you care about location or scale?

antman 4y ago

What do you use instead?

bigbillheck 4y ago
If I care about 'was this data generated by this specific distribution' then I'll quantize and do a chi-square. If I care about 'were these two datasets generated by the same distribution' then I'll step back and ask 'why do I care?' and 'what will I do with the answer?' and then use a more specific test.