4 ms·

One other thing which both this and PG's original theory get wrong: Their basic premise is wrong, if bias continues to exist after the selection event in quest

by lukev 11y ago

One other thing which both this and PG's original theory get wrong: Their basic premise is wrong, if bias continues to exist after the selection event in question. For example, if YC had (hypothetically) a real bias against black or women entrepreneurs, it is almost certain that future funding rounds, as well as all possible exit scenarios, would exhibit very much of the same bias. In which case, the future "performance" of those candidates would be poor, and by PG's definition unbiased even though the only meaningful result is that YC is no more biased than subsequent performance evaluations.

staunch 11y ago
This seems so obvious that it's possible we're missing something. It seems, at best, "A Way to Diff Your Bias", which isn't the same thing as detecting your bias at all. No one has the goal (I hope) of aligning their negative biases with others.
nepstein 11y ago
True - though this means only that the signal (outperformance by a group that is the target of discrimination) might not be present (if discrimination continues past initial selection as you point out). However, the test may still useful to help confirm bias. If outperformance is observed, you can infer one of 3 things is true: 1) there is bias at initial selection but not after (or at least reduced bias) 2) members of the outperforming group are simply stronger performers (different but still interesting) 3) there is no bias at selection but there are affirmative action effects after the initial selection (not obvious why this would be the case)
nshepperd 11y ago
The theory presumes a reliable way of measuring "actual performance"; that is, the quantity against which the selection process is supposedly biased. That's a limitation, but I wouldn't say it makes the test "wrong". It does mean that maybe monetary earnings or anything else sensitive to later-round bias are not the thing to use to measure candidate performance, at least if you're doing this for the social utility. Of course, if you're only in it to make money, and you're only in charge of the first round... then you really do want just an unbiased evaluation of the (biased) future earnings prospects. So in that case using raw earnings would be correct...
bjt 11y ago
It's reasonable to think that the bias would decrease over the company's lifetime. In early rounds there is little data on the company, so more decisions are made on hunches, and there's a lot of potential influence for bias. While there's potential for that in later rounds too, there's also a lot more objective information. The company is either making money or it isn't. The importance of relationships to the funding round also plays a role. If you get as far as an IPO, it seems unlikely that the stock-buying public is going to stay away because the founders are female/black/etc.
agarden 11y ago
Let's assume that bias does persist past selection through the duration of the program. Does that change the interpretation when you look at First Round Capital's data that shows its female founders outperforming the males by 63%? I don't think it does. The test may not be sufficient to prove that you have no bias, but it may be good enough to prove that you do. When it does indicate bias, it seems likely to be correct. To put it another way, if it is 1948 and the only three black people in Major League Baseball are all superstars, then the distribution of baseball skill among black players is extremely unbalanced or there is a lot of bias keeping the average and moderately-better-than-average black players out.
- notahacker 11y ago
  The interpretation is still somewhat unreliable as in many cases there's still another hypothesis, which is that $UNDERREPRESENTED_MINORITY actually overperforms due to favourable treatment after the selection process Of course favourable treatment can't make people into superstar startup founders or baseball players (and I'm sure any special treatment afforded to black baseball players in the 1940s was the complete opposite of favourable). But more generally it can make an organisation with fair selection processes look like it sets a higher bar for $MINORITY because it addresses low numbers by being very keen to promote and very reluctant to fire/deselect members of said minority, so these kind of studies still have to be considered with care. (Of course even if an organisation is proactively treating a minority group favourably after selection doesn't mean that conscious or unconscious biases don't exist in the selection process.)
- gozo 11y ago
  There's one thing I don't get though. Being biased often means valuing something that is common among those you favor, but rare among those you don't. So even if one group outperforms the other, why would that in a practical scenario prove that your next candidate should be of that group? It seems entirely possible that if your program is "narrow" enough you could exhaust the pool of more successful candidates from a minority group. Of course this would be a lot more plausible if you're biased.
hinkley 11y ago
It's almost as if they're trying to apply a technical solution to a social problem...
skj 11y ago
As uncomfortable as it may be too say, that means that it isn't bias on YC's part; it's an accurate predictor of future success. I would argue that social responsibility requires YC to take the hit, but bias is the wrong word if they don't.
yummyfajitas 11y ago
This isn't actually a flaw in the theory. The theory is designed to measure whether the selection process is inaccurately predicting outcomes. So an unbiased selection process will account for post-selection factors - including the fact that VCs might not like a person. It's also worth noting that an unbiased prediction is not necessarily "fair" in the colloquial sense. For example, I've seen data suggesting that an unbiased prediction of college outcomes would actually penalize black applicants, since black applicants underperform relative to their SATs and college grades. (The person who had this data was very careful not to draw this conclusion in the publication - career limiting move, as they say.) So a fair selection process which looks only at high school grades/SAT/etc might actually be biased as a statistical decision procedure.

by lukev 11y ago

staunch 11y ago

This seems so obvious that it's possible we're missing something. It seems, at best, "A Way to Diff Your Bias", which isn't the same thing as detecting your bias at all. No one has the goal (I hope) of aligning their negative biases with others.

nepstein 11y ago

True - though this means only that the signal (outperformance by a group that is the target of discrimination) might not be present (if discrimination continues past initial selection as you point out). However, the test may still useful to help confirm bias. If outperformance is observed, you can infer one of 3 things is true: 1) there is bias at initial selection but not after (or at least reduced bias) 2) members of the outperforming group are simply stronger performers (different but still interesting) 3) there is no bias at selection but there are affirmative action effects after the initial selection (not obvious why this would be the case)

nshepperd 11y ago

The theory presumes a reliable way of measuring "actual performance"; that is, the quantity against which the selection process is supposedly biased. That's a limitation, but I wouldn't say it makes the test "wrong". It does mean that maybe monetary earnings or anything else sensitive to later-round bias are not the thing to use to measure candidate performance, at least if you're doing this for the social utility. Of course, if you're only in it to make money, and you're only in charge of the first round... then you really do want just an unbiased evaluation of the (biased) future earnings prospects. So in that case using raw earnings would be correct...

bjt 11y ago

It's reasonable to think that the bias would decrease over the company's lifetime. In early rounds there is little data on the company, so more decisions are made on hunches, and there's a lot of potential influence for bias. While there's potential for that in later rounds too, there's also a lot more objective information. The company is either making money or it isn't. The importance of relationships to the funding round also plays a role. If you get as far as an IPO, it seems unlikely that the stock-buying public is going to stay away because the founders are female/black/etc.

agarden 11y ago

Let's assume that bias does persist past selection through the duration of the program. Does that change the interpretation when you look at First Round Capital's data that shows its female founders outperforming the males by 63%? I don't think it does. The test may not be sufficient to prove that you have no bias, but it may be good enough to prove that you do. When it does indicate bias, it seems likely to be correct. To put it another way, if it is 1948 and the only three black people in Major League Baseball are all superstars, then the distribution of baseball skill among black players is extremely unbalanced or there is a lot of bias keeping the average and moderately-better-than-average black players out.

notahacker 11y ago
The interpretation is still somewhat unreliable as in many cases there's still another hypothesis, which is that $UNDERREPRESENTED_MINORITY actually overperforms due to favourable treatment after the selection process Of course favourable treatment can't make people into superstar startup founders or baseball players (and I'm sure any special treatment afforded to black baseball players in the 1940s was the complete opposite of favourable). But more generally it can make an organisation with fair selection processes look like it sets a higher bar for $MINORITY because it addresses low numbers by being very keen to promote and very reluctant to fire/deselect members of said minority, so these kind of studies still have to be considered with care. (Of course even if an organisation is proactively treating a minority group favourably after selection doesn't mean that conscious or unconscious biases don't exist in the selection process.)
gozo 11y ago
There's one thing I don't get though. Being biased often means valuing something that is common among those you favor, but rare among those you don't. So even if one group outperforms the other, why would that in a practical scenario prove that your next candidate should be of that group? It seems entirely possible that if your program is "narrow" enough you could exhaust the pool of more successful candidates from a minority group. Of course this would be a lot more plausible if you're biased.

hinkley 11y ago

It's almost as if they're trying to apply a technical solution to a social problem...

skj 11y ago

As uncomfortable as it may be too say, that means that it isn't bias on YC's part; it's an accurate predictor of future success. I would argue that social responsibility requires YC to take the hit, but bias is the wrong word if they don't.

yummyfajitas 11y ago

This isn't actually a flaw in the theory. The theory is designed to measure whether the selection process is inaccurately predicting outcomes. So an unbiased selection process will account for post-selection factors - including the fact that VCs might not like a person. It's also worth noting that an unbiased prediction is not necessarily "fair" in the colloquial sense. For example, I've seen data suggesting that an unbiased prediction of college outcomes would actually penalize black applicants, since black applicants underperform relative to their SATs and college grades. (The person who had this data was very careful not to draw this conclusion in the publication - career limiting move, as they say.) So a fair selection process which looks only at high school grades/SAT/etc might actually be biased as a statistical decision procedure.