3 ms·

In a grandchild comment, you note that how close the min of each group is to the cutoff depends on sample size. This is true, but I think it's deal-breakingly t

In a grandchild comment, you note that how close the min of each group is to the cutoff depends on sample size. This is true, but I think it's deal-breakingly true. Suppose we have to input groups, A and B. Members of each group are distributed as Exp(1), ie same underlying distribution. Our selection procedure is totally fair as well: we take everyone without question. However, there are 9x as many people in group A as group B. So the min of group A accepted (= the min of all group A) will be distributed as Exp(9 * |B|) and the min of group B accepted will be distributed as Exp(|B|). So in expectation the min from group A will be smaller than the min from B, and indeed this happens 90% of the time. (Aren't exponentials nice?) Of course in this case we can note that this effect is from differences in sample size and exactly correct for it. But normally we will not know how to do this correction properly because we don't know the true underlying distribution or the acceptance criteria.

yummyfajitas 11y ago
First of all, making a group 9x larger doesn't change the shape of the distribution. The distribution would look like 9 exp(|x|), not exp(9|x|). Secondly, I calculate a p-value for this test here: https://news.ycombinator.com/item?id=10484309 https://news.ycombinator.com/item?id=10484309 It's proportional to the smaller of the sample sizes. I'll have a more detailed writeup soon. Also, this test is non-parametric, so you don't actually need to know either C or the distributions f and g - all you need is a certain level of uniform regularity in f and g.
- radioactivist 11y ago
  A note: The original post is correct in what it says. It is saying if one draws N independent samples from an exp(1) distribution then the minimum is distributed as exp(N). This is true. Because of this, for any finite sample there is a systematic 'bias' by your metric if the sample size differs between the two groups. However, you are correct in that this difference vanishes as N becomes large and further that one can make controlled statements about the expected size of this difference, given the distribution.
  - yummyfajitas 11y ago
    Oops - I did misread what he wrote. He was discussing the distribution of the min of the sample, not the distribution itself. Also, there may be a bias in the true p-value, but there is none in the p-value bound based on h(d) and min(N1, N2).

yummyfajitas 11y ago

First of all, making a group 9x larger doesn't change the shape of the distribution. The distribution would look like 9 exp(|x|), not exp(9|x|). Secondly, I calculate a p-value for this test here: https://news.ycombinator.com/item?id=10484309 https://news.ycombinator.com/item?id=10484309 It's proportional to the smaller of the sample sizes. I'll have a more detailed writeup soon. Also, this test is non-parametric, so you don't actually need to know either C or the distributions f and g - all you need is a certain level of uniform regularity in f and g.

radioactivist 11y ago
A note: The original post is correct in what it says. It is saying if one draws N independent samples from an exp(1) distribution then the minimum is distributed as exp(N). This is true. Because of this, for any finite sample there is a systematic 'bias' by your metric if the sample size differs between the two groups. However, you are correct in that this difference vanishes as N becomes large and further that one can make controlled statements about the expected size of this difference, given the distribution.
- yummyfajitas 11y ago
  Oops - I did misread what he wrote. He was discussing the distribution of the min of the sample, not the distribution itself. Also, there may be a bias in the true p-value, but there is none in the p-value bound based on h(d) and min(N1, N2).