5 ms·

People, most of whom clearly are not that good at math, are being really harsh on Paul Graham. Graham is mostly right, but slightly incorrect. In particular, s

People, most of whom clearly are not that good at math, are being really harsh on Paul Graham. Graham is mostly right, but slightly incorrect. In particular, suppose group A has the distribution f(x) and B has the distribution g(x). If f(x) and g(x) are shaped significantly differently past the cutoff, then mean(H(x-C)f(x)) and mean(H(x-c)g(x)) might not agree even though there is no bias by construction. (Here H(x) is a step function and C the cutoff). However, there is an easy fix: compute the minima of the support of the distribution rather than mean. min(H(x-C)f(x)) = min(H(x-C)g(x)) = C. In practice, measure the weakest male and weakest female to be accepted in your sample set, or some similar approximation. I'm pretty sure this is a valid frequentist hypothesis test. I've got half a proof worked out on paper already. It depends very weakly (and non-parametrically) on f(x) and g(x), but it works in basically the exact way Graham wants it to. Every counterexample I can think of is really pathological. My next blog post will probably be a proof of this. All this negativity is really an overreaction. I know it's fun to totally debunk someone on details, but these are mostly fixable details.

MattHeard 11y ago
Wouldn't the weakest male and weakest female founders be underperforming more due to individual factors than bias? Maybe a mean of the lower quartile would remove some the the noise.
- yummyfajitas 11y ago
  Suppose the cutoff for men is C but for women is C+K. Then the weakest man can be expected to have quality C+epsilon, while the weakest woman will have quality C+K+epsilon. Here epsilon is how close a typical candidate will be to the cutoff, and is mainly a function of the sample size. I don't know the behavior of epsilon off the top of my head.
  - tprice7 11y ago
    I think MattHeard understood this and his point was that the weakest man will realistically have quality C + epsilon + random, and the weakest woman will have quality C + K + epsilon + random. The random term arises because no evaluation process is going to perfectly tell you how people are going to end up performing. But yeah, this seems fixable also, by averaging some number of the lowest performing members of each group, as MattHeard suggested.
    - yummyfajitas 11y ago
      I'm not 100% sure how to deal with noise + extremal statistics. But I've got a 12 hour plane ride ahead of me tomorrow, so I can probably work out a fix.
      - grayclhn 11y ago
        I suspect the math is easier if you reframe the hypothesis to be about the derivatives: adding one more person from either group should have the same marginal effect on outcomes. As you get more observations you get more in a neighborhood of the cutoff which gives a consistent estimator of the derivative.
quietplatypus 11y ago
Isn't this not as meaningful as the mean, because the minimum doesnt really tell you about true capability? I can also easily construct an institution that satisfies your parameters, the minimums of two groups match, as in they have a token member at C for every group, but the characteristic performance of all members is different? This in my eyes amounts to a simplification that is neat mathematically, but removes most of the useful information. I admit though that if the minimums did not match very well, that is an obvious sign of bias. Edit: I suppose what I'm essentially saying is, what if minimums do match, then we cannot rule out bias.
- yummyfajitas 11y ago
  If you think you can construct examples that break my test, go ahead and do it. I'm curious to see what you can come up with. However, your claim that you can have "a token member at C for every group" is merely the claim that adversarial sampling can destroy any statistical procedure. So what? This is boring mathematically. Nor is it relevant to the problem at hand unless you want to claim that First Round Capital is actively conspiring to both a) be biased and earn less money and b) waste more money hiding that bias by funding tokens.
SamReidHughes 11y ago
> In practice, measure the weakest male and weakest female to be accepted in your sample set, or some similar approximation. In practice, where do you get these definitive unbiased measurements of the expected return of investing in an individual (or graduation probability or some other score)?
- morgante 11y ago
  This test can be used after the fact, so you're not measuring expected return but actual return. It's pretty easy to measure actual return (or actual graduation rate, or actual GPA, etc.).
  - SamReidHughes 11y ago
    What test? We're talking about computing the minimum of what before-the-fact EV should be (based on what set of information?) here, not computing what the mean EV should be (for which you can add up the outcomes and divide by N).
    - morgante 11y ago
      > What test? The test for bias proposed by pg and modified by yummyfajitas. > before-the-fact EV Why are you bringing in before-the-fact EV at all? That's not a component. The test is based on comparing the minimums, yes. So, for example, what is GPA of the worst female and male students. What part of that requires expectations?
      - SamReidHughes 11y ago
        Before-the-fact EV is what the entire question of bias is about. In the case of investments, your minimum outcome is that you shut down the company. Comparing this for male/female sets is useless. Also, looking at other aspects of the distribution (e.g. how many $10M exits, $100M exits) isn't useful because for investing, you want to maximize mean return, and it's not like looking at 20-80%tile outcomes numerically tells you much about that -- are these people saving dead companies or are they derisking $10B companies? With students there is more of a connection between before-the-fact EV and outcomes, and much more information exposed (grades and course registrations on a semester-by-semester basis). Without that information (if you just looked at who graduated, who didn't) you can't really say as much. (You could run a Netflix-like competition to see who has the best graduation prediction engine and use its estimates as your definitive answer to what an unbiased EV predictor would say about college applicants.) With GPA information, let's say you look at the 10th percentile GPA's in each set and below, for boys it's 0.1-1.1 and for girls it's 0.1-1.5. What does that tell you about before-the-fact EV cutoffs? It doesn't tell you much, because one set of students, when they fail, could be more likely to fail harder than the other. It takes a lot more work than just that.
LukaAl 11y ago
I like your proposal in theory but I doubt it works in practice. I'll try to explain: in practice what First Round has done is measuring post investment performance. I'm pretty sure that if the sample is big enough you will have an "all male" startup and a "with female" startup that failed thus the min being zero in both case. This doesn't mean there's no bias. The point is that when you decide which company you finance (the source of bias) you make an estimation of future potential. But your test (and PG's one) measure ex-post results. Since there's a lot of uncertainty between the ex-ante and the ex-post measure, your test doesn't work. Let me put in another way. The measure you are able to perform is not H(x-C)f(x) but a * H(x-C)f(x) + (1-a) * random where random is a random number and a is the weight of f(x) on the final outcome. You are right if a is 1, you and pg are wrong if a is 0 (but it will be a problem for the VCs), for everything in between you have to make assumption on the distribution of random.
- yummyfajitas 11y ago
  I discuss in a different comment the effect of additive noise. I admit I haven't worked out the details of the min statistic and additive noise yet. But I'm actually fairly bullish on it - this problem is mostly just simple statistics. I also suspect that first rounds data is useless. PG is being too credible about it - I suspect he is just being political and making his post more PC, since he doesnt seem that statistically naive. (Taking a lesson from Sam Altman most likely.)
Tloewald 11y ago
There's also the fact that the effect size is huge — 63%. Most real world population differences are not weird enough to generate that large an effect without bias. The bigger questions are how much would the effect size change without removing outliers (Uber as pg mentions but presumably others too) and how significant is the result.
Tarrosion 11y ago
In a grandchild comment, you note that how close the min of each group is to the cutoff depends on sample size. This is true, but I think it's deal-breakingly true. Suppose we have to input groups, A and B. Members of each group are distributed as Exp(1), ie same underlying distribution. Our selection procedure is totally fair as well: we take everyone without question. However, there are 9x as many people in group A as group B. So the min of group A accepted (= the min of all group A) will be distributed as Exp(9 * |B|) and the min of group B accepted will be distributed as Exp(|B|). So in expectation the min from group A will be smaller than the min from B, and indeed this happens 90% of the time. (Aren't exponentials nice?) Of course in this case we can note that this effect is from differences in sample size and exactly correct for it. But normally we will not know how to do this correction properly because we don't know the true underlying distribution or the acceptance criteria.
- yummyfajitas 11y ago
  First of all, making a group 9x larger doesn't change the shape of the distribution. The distribution would look like 9 exp(|x|), not exp(9|x|). Secondly, I calculate a p-value for this test here: https://news.ycombinator.com/item?id=10484309 https://news.ycombinator.com/item?id=10484309 It's proportional to the smaller of the sample sizes. I'll have a more detailed writeup soon. Also, this test is non-parametric, so you don't actually need to know either C or the distributions f and g - all you need is a certain level of uniform regularity in f and g.
  - radioactivist 11y ago
    A note: The original post is correct in what it says. It is saying if one draws N independent samples from an exp(1) distribution then the minimum is distributed as exp(N). This is true. Because of this, for any finite sample there is a systematic 'bias' by your metric if the sample size differs between the two groups. However, you are correct in that this difference vanishes as N becomes large and further that one can make controlled statements about the expected size of this difference, given the distribution.
    - 11y ago

MattHeard 11y ago

Wouldn't the weakest male and weakest female founders be underperforming more due to individual factors than bias? Maybe a mean of the lower quartile would remove some the the noise.

yummyfajitas 11y ago
Suppose the cutoff for men is C but for women is C+K. Then the weakest man can be expected to have quality C+epsilon, while the weakest woman will have quality C+K+epsilon. Here epsilon is how close a typical candidate will be to the cutoff, and is mainly a function of the sample size. I don't know the behavior of epsilon off the top of my head.
- tprice7 11y ago
  I think MattHeard understood this and his point was that the weakest man will realistically have quality C + epsilon + random, and the weakest woman will have quality C + K + epsilon + random. The random term arises because no evaluation process is going to perfectly tell you how people are going to end up performing. But yeah, this seems fixable also, by averaging some number of the lowest performing members of each group, as MattHeard suggested.
  - yummyfajitas 11y ago
    I'm not 100% sure how to deal with noise + extremal statistics. But I've got a 12 hour plane ride ahead of me tomorrow, so I can probably work out a fix.
    - grayclhn 11y ago
      I suspect the math is easier if you reframe the hypothesis to be about the derivatives: adding one more person from either group should have the same marginal effect on outcomes. As you get more observations you get more in a neighborhood of the cutoff which gives a consistent estimator of the derivative.

quietplatypus 11y ago

Isn't this not as meaningful as the mean, because the minimum doesnt really tell you about true capability? I can also easily construct an institution that satisfies your parameters, the minimums of two groups match, as in they have a token member at C for every group, but the characteristic performance of all members is different? This in my eyes amounts to a simplification that is neat mathematically, but removes most of the useful information. I admit though that if the minimums did not match very well, that is an obvious sign of bias. Edit: I suppose what I'm essentially saying is, what if minimums do match, then we cannot rule out bias.

yummyfajitas 11y ago
If you think you can construct examples that break my test, go ahead and do it. I'm curious to see what you can come up with. However, your claim that you can have "a token member at C for every group" is merely the claim that adversarial sampling can destroy any statistical procedure. So what? This is boring mathematically. Nor is it relevant to the problem at hand unless you want to claim that First Round Capital is actively conspiring to both a) be biased and earn less money and b) waste more money hiding that bias by funding tokens.

SamReidHughes 11y ago

> In practice, measure the weakest male and weakest female to be accepted in your sample set, or some similar approximation. In practice, where do you get these definitive unbiased measurements of the expected return of investing in an individual (or graduation probability or some other score)?

morgante 11y ago
This test can be used after the fact, so you're not measuring expected return but actual return. It's pretty easy to measure actual return (or actual graduation rate, or actual GPA, etc.).
- SamReidHughes 11y ago
  What test? We're talking about computing the minimum of what before-the-fact EV should be (based on what set of information?) here, not computing what the mean EV should be (for which you can add up the outcomes and divide by N).
  - morgante 11y ago
    > What test? The test for bias proposed by pg and modified by yummyfajitas. > before-the-fact EV Why are you bringing in before-the-fact EV at all? That's not a component. The test is based on comparing the minimums, yes. So, for example, what is GPA of the worst female and male students. What part of that requires expectations?
    - SamReidHughes 11y ago
      Before-the-fact EV is what the entire question of bias is about. In the case of investments, your minimum outcome is that you shut down the company. Comparing this for male/female sets is useless. Also, looking at other aspects of the distribution (e.g. how many $10M exits, $100M exits) isn't useful because for investing, you want to maximize mean return, and it's not like looking at 20-80%tile outcomes numerically tells you much about that -- are these people saving dead companies or are they derisking $10B companies? With students there is more of a connection between before-the-fact EV and outcomes, and much more information exposed (grades and course registrations on a semester-by-semester basis). Without that information (if you just looked at who graduated, who didn't) you can't really say as much. (You could run a Netflix-like competition to see who has the best graduation prediction engine and use its estimates as your definitive answer to what an unbiased EV predictor would say about college applicants.) With GPA information, let's say you look at the 10th percentile GPA's in each set and below, for boys it's 0.1-1.1 and for girls it's 0.1-1.5. What does that tell you about before-the-fact EV cutoffs? It doesn't tell you much, because one set of students, when they fail, could be more likely to fail harder than the other. It takes a lot more work than just that.

LukaAl 11y ago

I like your proposal in theory but I doubt it works in practice. I'll try to explain: in practice what First Round has done is measuring post investment performance. I'm pretty sure that if the sample is big enough you will have an "all male" startup and a "with female" startup that failed thus the min being zero in both case. This doesn't mean there's no bias. The point is that when you decide which company you finance (the source of bias) you make an estimation of future potential. But your test (and PG's one) measure ex-post results. Since there's a lot of uncertainty between the ex-ante and the ex-post measure, your test doesn't work. Let me put in another way. The measure you are able to perform is not H(x-C)f(x) but a * H(x-C)f(x) + (1-a) * random where random is a random number and a is the weight of f(x) on the final outcome. You are right if a is 1, you and pg are wrong if a is 0 (but it will be a problem for the VCs), for everything in between you have to make assumption on the distribution of random.

yummyfajitas 11y ago
I discuss in a different comment the effect of additive noise. I admit I haven't worked out the details of the min statistic and additive noise yet. But I'm actually fairly bullish on it - this problem is mostly just simple statistics. I also suspect that first rounds data is useless. PG is being too credible about it - I suspect he is just being political and making his post more PC, since he doesnt seem that statistically naive. (Taking a lesson from Sam Altman most likely.)

Tloewald 11y ago

There's also the fact that the effect size is huge — 63%. Most real world population differences are not weird enough to generate that large an effect without bias. The bigger questions are how much would the effect size change without removing outliers (Uber as pg mentions but presumably others too) and how significant is the result.

Tarrosion 11y ago

In a grandchild comment, you note that how close the min of each group is to the cutoff depends on sample size. This is true, but I think it's deal-breakingly true. Suppose we have to input groups, A and B. Members of each group are distributed as Exp(1), ie same underlying distribution. Our selection procedure is totally fair as well: we take everyone without question. However, there are 9x as many people in group A as group B. So the min of group A accepted (= the min of all group A) will be distributed as Exp(9 * |B|) and the min of group B accepted will be distributed as Exp(|B|). So in expectation the min from group A will be smaller than the min from B, and indeed this happens 90% of the time. (Aren't exponentials nice?) Of course in this case we can note that this effect is from differences in sample size and exactly correct for it. But normally we will not know how to do this correction properly because we don't know the true underlying distribution or the acceptance criteria.

yummyfajitas 11y ago
First of all, making a group 9x larger doesn't change the shape of the distribution. The distribution would look like 9 exp(|x|), not exp(9|x|). Secondly, I calculate a p-value for this test here: https://news.ycombinator.com/item?id=10484309 https://news.ycombinator.com/item?id=10484309 It's proportional to the smaller of the sample sizes. I'll have a more detailed writeup soon. Also, this test is non-parametric, so you don't actually need to know either C or the distributions f and g - all you need is a certain level of uniform regularity in f and g.
- radioactivist 11y ago
  A note: The original post is correct in what it says. It is saying if one draws N independent samples from an exp(1) distribution then the minimum is distributed as exp(N). This is true. Because of this, for any finite sample there is a systematic 'bias' by your metric if the sample size differs between the two groups. However, you are correct in that this difference vanishes as N becomes large and further that one can make controlled statements about the expected size of this difference, given the distribution.
  - 11y ago