5 ms·

Going down this rabbit hole eventually leads you to nonparametric statistical tests, e.g. Mann-Whitney-U and so on.

cperciva 8y ago
Right. And those are also more powerful (and don't need any knowledge of the distribution of measurement errors). I mentioned the "three old and three new" test because it's simple, not because it's powerful.
- blt 8y ago
  more powerful in the statistical sense? I though non-parametric tests were usually less powerful than those where a certain distribution is assumed.
  - adamc 8y ago
    That depends on whether the assumed error distribution is accurate. Obviously, if the distribution is known, you will do better by incorporating it. But assuming Gaussian errors when the data doesn't match can lead to bad analyses.
  - srean 8y ago
    You would be surprised what number of samples does to the tests. Take t-test -- the most powerful test to check if means of 2 equivariant Gaussians differ. If you compare the asymptotic efficiency of Mann-Whiney (a distribution free test) relative to t-test is around 0.96. Of course in practice you will not have infinite samples. It then comes down when do these asymptotics kick in. Unfortunately that depends on the distribution.
yen223 8y ago
My personal favourite quick-and-dirty trick: A quick way to estimate any distribution's median is to draw 5 random samples. There's a >90% chance that the median is between the biggest and the smallest value.
- AstralStorm 8y ago
  The final bit is complete bunkum for an unknown distribution. Whether the distribution is monotonic and if not, how symmetric, is the major determinant of occurrence of such result. Giving a flat number is completely bogus.
  - yen223 8y ago
    I don't understand what you mean. It doesn't actually matter what shape the distribution takes. That figure is derived entirely from what the "median" means, i.e. a random sample has 50% chance of being greater than the median, and 50% chance of being lower than it. A bit of maths would show that the probability that all 5 samples lie entirely above or entirely below the median (i.e. the median is NOT between the greatest and the smallest value) is 1/16. That gives you a 15/16 (= 93.75%) chance that the median is contained within the bounds.

Right. And those are also more powerful (and don't need any knowledge of the distribution of measurement errors). I mentioned the "three old and three new" test because it's simple, not because it's powerful.

blt 8y ago
more powerful in the statistical sense? I though non-parametric tests were usually less powerful than those where a certain distribution is assumed.
- adamc 8y ago
  That depends on whether the assumed error distribution is accurate. Obviously, if the distribution is known, you will do better by incorporating it. But assuming Gaussian errors when the data doesn't match can lead to bad analyses.
- srean 8y ago
  You would be surprised what number of samples does to the tests. Take t-test -- the most powerful test to check if means of 2 equivariant Gaussians differ. If you compare the asymptotic efficiency of Mann-Whiney (a distribution free test) relative to t-test is around 0.96. Of course in practice you will not have infinite samples. It then comes down when do these asymptotics kick in. Unfortunately that depends on the distribution.

yen223 8y ago

My personal favourite quick-and-dirty trick: A quick way to estimate any distribution's median is to draw 5 random samples. There's a >90% chance that the median is between the biggest and the smallest value.

AstralStorm 8y ago
The final bit is complete bunkum for an unknown distribution. Whether the distribution is monotonic and if not, how symmetric, is the major determinant of occurrence of such result. Giving a flat number is completely bogus.
- yen223 8y ago
  I don't understand what you mean. It doesn't actually matter what shape the distribution takes. That figure is derived entirely from what the "median" means, i.e. a random sample has 50% chance of being greater than the median, and 50% chance of being lower than it. A bit of maths would show that the probability that all 5 samples lie entirely above or entirely below the median (i.e. the median is NOT between the greatest and the smallest value) is 1/16. That gives you a 15/16 (= 93.75%) chance that the median is contained within the bounds.