Yeah true dat. Still, if you have 5-50 samples from an unknown distribution I think the KS test can be a very valuable protection against false-positive i.e. s

Yeah true dat. Still, if you have 5-50 samples from an unknown distribution I think the KS test can be a very valuable protection against false-positive i.e. seeing patterns in random noise. Even more so if you are making multiple comparisons, e.g. comparing multiple response variables, since you can easily make a Bonferroni correction to avoid "the xkcd 882 problem" which is hard to do by eyeball. So I see a lot of utility in the KS test (and Wilcox, etc) when measuring computer systems on multiple metrics with unknown distributions (often not Gaussian) and where samples cost ~$0.10-$1.0 each. I see a lot of false positives in the absence of such tests, or otherwise generally low confidence in inferences based on e.g. eyeball/mean/median.