11 ms·

On a simple mathematical basis, this is false. Consider two groups of candidates for a scholarship, A and B. We want to select all candidates that have an 80%

On a simple mathematical basis, this is false. Consider two groups of candidates for a scholarship, A and B. We want to select all candidates that have an 80% or better chance of graduation. Group A comes from a population where the chance of graduation is distributed uniformly from 0% to 100% and group B is from one where the chance is distributed uniformly from 10% to 90%, with the same average but less variation in group B. Now suppose that we select without bias or inaccuracy all the applicants that have an 80% or better chance of graduation. That means we select a subset of A with a range of 80% to 100% and a subset of B with a range from 80% to 90%. The average graduation rate of scholarship winners from group A will be 90% and that from group B will be 85%. But we haven't been biased against A. We've selected according to the exact same perfect evaluation process and criterion from both groups. It was just their prior distribution that was different. The actual applicant groups for jobs or financing in the real world, when they are divided by demographic factors like age, sex, race, and educational level, will almost always manifest different variances in success levels even when the averages are the same. That makes this test useless and mathematically illiterate. And when we use a normal distribution, as we should always expect given the central limit theorem, the mathematical problems get even more intense. This short comment is not up to pg's usual high standards for his essays.

eoi 11y ago
The article defines bias as follows: > Want to know if the selection process was biased against some type of applicant? Check whether they outperform the others. This is not just a heuristic for detecting bias. It's what bias means. Under that definition, you have been biased against A. [edit: on reflection I see this as a weakness of his definition. I missed that your selection process does in fact select the best candidates.]
- bjwbell 11y ago
  Yes, but that's not the common usage of the word. Or how most people understand it. With that usage you could say ivy league schools are NOT biased against Asians since Asian graduates aren't more successful than non-Asian ones, except nobody does.
  - 1457389 11y ago
    Hypothetical logic is flawed anyway. Higher ability Asian graduates could be less / only equally successful in the workplace due to pervasive external bias too. A lot of this is exacerbated by the fact that "soft skills" are more important for high status careers, and your "soft skills" are pretty much defined by tribal associations. It's the core of how we interact socially, and it causes problems that are really only fixed by alleviating scarcity.
- decisiveness 11y ago
  Unless you know what exactly caused A to outperform others, you won't really know if the process is biased or what made it biased. When asserting biases, you must first distinguish them from random noise. Using pg's logic, every selection process that isn't perfect is biased.
- azakai 11y ago
  >> This is not just a heuristic for detecting bias. It's what bias means. > Under that definition That's not a definition. It's a claim about what the term "bias" means.
suchow 11y ago
This logic is similar to that of Larry Summers in his remarks about diversity in science and engineering, http://www.harvard.edu/president/speeches/summers_2005/nber.php http://www.harvard.edu/president/speeches/summers_2005/nber.....
rewqfdsa 11y ago
Why are you using biased mathematics? If statistics and the scientific method (the tools through which dead white men continue to colonize) give us obviously problematic results, we should abandon them in favor of a method of inquiry that promotes social justice.
- WildUtah 11y ago
  I enjoyed this comment.
- SFLemonade 11y ago
  Please tell me this is sarcasm...
  - tprice7 11y ago
    Based on rewqfdsa's comment history, this is probably sarcasm.
- graycat 11y ago
  > social justice Gee, I never saw a definition. Not sure the meaning of the phrase is clear without a definition.
robot22 11y ago
I find it slightly amazing he posted this. Did he not bother to ask someone with a probabilistic background before posting?
yummyfajitas 11y ago
Graham's intuition is assuming equality of the two distributions. As I noted in a different comment here, you can pretty easily fix Graham's test. Compute min(accepted a) and min(accepted B) instead of the means. In your example, the min of the accepted distributions would both work out to be 80%.
- tansey 11y ago
  So the PG estimator is clearly problematic. I agree that the yummfajitas (YM) estimator looks to be consistent. In this case though, we're dealing with (small) finite sample sizes, so we need to come up with some sort of test statistic. What would the YM test be here? It seems tricky since you are dealing with a conditional distribution based on left-censored data. I'm also not aware of any difference-of-minimums test, though I am happy to be educated if there is one!
  - yummyfajitas 11y ago
    I don't know of something to refer to, but I don't think the statistics are too hard. The test statistic would be exactly min(sample1) and min(sample2). Suppose the cutoff sample is distributed according to f(x)H(x-C). Then the probability of the minima of a sample exceeding C+e by random chance, assuming the null hypothesis, is p = (1-\int_C^{C+e}f(x) dx)^N. So now you have a frequentist hypothesis test. If you make reasonable assumptions on f(x) (non-vanishing near C, quantified somehow), it's even nice and non-parametric.
    - tansey 11y ago
      Does that assume both samples are identically distributed and the only difference is the cutoff? If it does, then couldn't we just continue to do a difference of means test and still be consistent? If it doesn't, how do you handle identifying the cutoff minima and the two different distributions in a frequentist way?
      - yummyfajitas 11y ago
ChicagoBoy11 11y ago
Great insight. I had the great fortune to take a course with Gary Becker where we went into some of the mathematics of college admissions. He made this precise point -- the variances of the particular populations you are looking at matter a great detail. He managed to build some pretty convincing models which provided a compelling narrative for "biases" that we seemed to observe in the real world, all with simple changes to the distributions of populations. Great comment.
- WildUtah 11y ago
  I had the great fortune to take a course with Gary Becker Lucky you. He was great. I never had the chance to take a class from him but it would have been worth a Chicago winter to have the chance.
  - ChicagoBoy11 11y ago
    Second-best: https://www.youtube.com/watch?v=QajILZ3S2RE&list=PL9334868E7A821E2A https://www.youtube.com/watch?v=QajILZ3S2RE&list=PL9334868E7... (I took this very class the year prior)
mattmaroon 11y ago
So in this particular case, assuming that First Round's sample size is significant, it may just be that the female founders who seek them out are just on average better than the male ones? I suppose that if women think that the selection process is biased against them, and most do (and it may be) perhaps the less than excellent ones just don't apply, whereas that isn't true for males?
- deleted 11y ago
  [deleted]
- rattray 11y ago
  This seems plausible to me (though no less problematic, of course).
- Stasis5001 11y ago
  The problem is different than that. It's that the measurement of performance is susceptible to differences in the conditional distribution of disparate groups, given that they were selected. We can illustrate it by modifing the GP example to not include time at all, and make the metric perfect. Suppose we are selecting for the next qualifying round for the Olympic 400m team, and select candidates if their 400m is under 50 seconds. Then, we measure performance -- immediately -- by a 400m run. We have two candidate pools: people who compete in the professional circuit, and everybody else. 100% of the former group who apply qualify, while only 10% of the others do. Okay, so now we immediately measure the average 400m time of all professionals, versus average time of all amateurs who can beat 50 seconds. It's pretty reasonable to expect that the professionals might average closer to 45 seconds, while the other group might average around 48 (I'm not a 400m expert, the actual numbers might be off. WR is 43 seconds). According to the article, we now conclude that our selection process is actually biased _against_ professionals! This is at the least very counter-intuitive. Maybe we provide both groups with coaching, and re-test after 6 months, a year, whatever. The professionals will certainly still outperform the amateurs. However, suppose one of the amateurs goes on to greatness, and wins. Wouldn't this obviously be biased against amateurs, according to our intuition?
- devalier 11y ago
quietplatypus 11y ago
Very interesting! Seems like I don't understand probability as well as I think I do, since I bought the argument till I ran into this comment. So, what are the effects of variance in different evaluation contexts, and do we have a meaningful way to measure bias if we take variance into account? My initial reactions: - It seems the higher the variance, the better examples you can trot out to say you are not biased against that particular group. since youll always find a member of that group who does amazing. - If distributions of performance are multimodal, its even harder to conclude stuff because different institutions might cut off different modes when selecting the bar. - Modeling the sources of variance may lead to insight into any actual bias.
hibikir 11y ago
The problem here is language and what our actual objectives are. When people complain about bias, they are not really talking about mathematical bias, but about something else: Their idea of fairness. They are talking about discrimination. And when we are discussing that, we can't really think about whether rules are applied fairly or not, but whether the rules produce the outcomes that we want. Let's go for a ludicrous example: We'll accept all applicants whose IQ is higher than their weight in pounds. We'll be explicitly discriminating against heavy people, but at the same time, we have pretty clear implicit biases against men, and ethnic groups who tend to be taller. We might as well have said that we prefer children and Japanese women. There's no need for mathematical bias: The bias comes from the rule selection. So, in your example, if our actual objective is to graduate an even amount of people from groups A and B, we have to, explicitly, make it easier for group B to get the scholarship. And many times organizations have objectives like that. As a more real example, let's consider a police department. If the objective is to have a racial makeup that represents the community, and different races have different drop-out rates, the candidate selection will prefer one kind over the other, precisely to counter the drop-out differential. So when regular people, and not mathematicians, discuss bias, the mathematical definition is unimportant. The one important thing is our stated objectives.
- 55555 11y ago
  Great comment. There are two types of fairness, (a) fair rules, and (b) fair outcome.
  - rewqfdsa 11y ago
    Favoring fair outcome over having fair rules is against everything I believe. We should not strive for a participation trophy culture.
    - lexcorvus 11y ago
      Indeed. The problem is that people frequently infer unfair rules from unequal outcomes, without taking into account the possibility of systematic group differences. Alan: I believe in equality of opportunity, not equality of outcome. Bob: How do you know there isn't equality of opportunity? Alan: Well, just look at how unequal the outcomes are! At this point, Bob would be wise to change the subject, because if he pressed on, he might get this: Bob: Can you give me an example? Alan: Group X is underrepresented in Field Y. Bob: Maybe Group X isn't as good at Field Y. Alan: What? That's racist and/or sexist! And if Bob were to make a comment to this effect on Hacker News, he's probably get downvoted. This is because most people agree with Alan, and many of them abuse their downvote privileges to punish ideas they disagree with rather than those that don't further the discussion. This degrades the quality of discourse, but at least it helps reassure the downvoters that they aren't racist and/or sexist.
pg 11y ago
It's true that this test assumes groups of applicants are roughly equal in (distribution of) ability. That is the default assumption in most conversations I've been involved in about bias, and particularly the example I used, but I'll add something making that explicit.
- gleb 11y ago
  I like the idea, but how do you apply this to power law distribution outcomes and get any statistical significance? I don't know the answer. E.g. the underlying First Round's analysis likely has no statistical significance. Assuming the power law distribution of outcomes top 5 outcomes will account for 97% of value. So we now have a study with n=5. To make the point let's apply this to YC's own portfolio. Assuming Dropbox, AirBnb and Stripe represent 75% of its value, we'll learn that YC is incredibly biased against: * MIT graduates * brother founders * founding teams that do not have female founders * and especially males named Drew Hard to believe these conclusions are correct or actionable
  - graycat 11y ago
    > I like the idea, but how do you apply this to power law distribution outcomes and get any statistical significance? I don't know the answer. See my post https://news.ycombinator.com/item?id=10484602 https://news.ycombinator.com/item?id=10484602 where are distribution-free. So "power law", Gaussian, anything else, doesn't matter.
  - deleted 11y ago
    [deleted]
- compbio 11y ago
  If feel the addition: "C" the applicants you're looking at have roughly equal distribution of ability. makes the reasoning more tautological/weak. If we take two dart boards (one for female -, one for male founders) as a visual, where hitting near the bull's eye counts as "startup success". If we take "C" to be true, then the darts would be thrown at random. Now we draw a circle around the bull's eye. Anything landing in this circle we fund. If this circle has a smaller radius on the female dartboard, than on the male dartboard, then evidently the smaller female circle will contain more darts closer to the target (better average performance) than the larger radius male circle. But then we do not even need performance numbers: Smaller radius circles will have less darts in them. Using "C" we only need to know that the male-female accept ratio is not 50%-50% for us to have found a bias. In short: If you see a roughly equal distribution of ability, and (for simplicity) a roughly equal number of female to male fundraisers, then you should always have a roughly equal distribution of female to male founders in your portfolio, performance be damned. The technique is still useful for when you do not have these female vs. male accept ratio's, and a VC publishes only success rates, but this information on ratio's is often more public than success rates/estimates.
vog 11y ago
> But we haven't been biased against A. [...] It was just their prior distribution that was different. This is why that isn't a counter example. Right at the beginning, the article clearly states that this method is only applicable if the prior distribution is equal: | You can use this technique whenever (a) [...], (b) [...], and (c) the groups of applicants you're looking at have roughly equal distribution of ability.
- zodiac 11y ago
  AFAIK pg added that to the article after WildUtah made his comment, and pg acknowledges this in a reply to WildUtah
jdoliner 11y ago
I think there's a big problem with this counter example. Which is that you're selecting for one thing and judging whether or not you were biased based on another. > Now suppose that we select without bias or inaccuracy all the applicants that have an 80% or better chance of graduation. This is subtly different than selecting for the highest graduation rate possible because it's binary, you want a group with >80% chances not a group with the best chances. Imagine if instead of the distributions you had group A was composed of people with a 100% chance of graduation and B was composed of people with an 80% chance of graduation. Our process does nothing to distinguish between those people because that extra 20% chance of graduation doesn't matter. This brings me to what I think is the fundamental problem with your criticism, it's not clear to me what it means for a group in your example to over perform. If you select a group with the goal of 80% of them graduating it doesn't make sense to call 90% of them graduating an over performance. That only makes sense if your goal up front is to maximize the graduation rate. I think if you rerun your example but instead assume an unbiased strategy that selects for the highest graduation rate possible you'll find that pg's essay makes a lot more sense.
meowface 11y ago
I suppose PG really should have said: >the groups of applicants you're looking at have exactly equal distribution of ability. Rather than "roughly equal". But obviously that makes the whole thing infeasible.
sdegutis 11y ago
> This short comment is not up to pg's usual high standards for his essays. I can almost here him thinking in response, "if I throw a dog a bone, I don't want to know if it tastes good or not."
dalek2point3 11y ago
absolutely spot on. Differences in distribution is only one way in which you could disprove pg. There are others. For example, different "treatment effects". If conditional on getting selected, VCs pay more attention or are more useful for women, then that would be another reason that we would get the pattern pg proposes, but is not due to bias at selection.
learnstats2 11y ago
Your premise discriminates between two groups. It's not clear how you can assign a candidate a 90% chance of graduation. That probability must be a subjective assessment that has come from some (biased) source. In truth, an individual will either graduate or not. In your example, you can assign 0% and 100% probabilities in group A, but you can't in group B. The most plausible mathematical explanation for that is that you collected insufficient relevant information about candidates in group B.

eoi 11y ago

The article defines bias as follows: > Want to know if the selection process was biased against some type of applicant? Check whether they outperform the others. This is not just a heuristic for detecting bias. It's what bias means. Under that definition, you have been biased against A. [edit: on reflection I see this as a weakness of his definition. I missed that your selection process does in fact select the best candidates.]

bjwbell 11y ago
Yes, but that's not the common usage of the word. Or how most people understand it. With that usage you could say ivy league schools are NOT biased against Asians since Asian graduates aren't more successful than non-Asian ones, except nobody does.
- 1457389 11y ago
  Hypothetical logic is flawed anyway. Higher ability Asian graduates could be less / only equally successful in the workplace due to pervasive external bias too. A lot of this is exacerbated by the fact that "soft skills" are more important for high status careers, and your "soft skills" are pretty much defined by tribal associations. It's the core of how we interact socially, and it causes problems that are really only fixed by alleviating scarcity.
decisiveness 11y ago
Unless you know what exactly caused A to outperform others, you won't really know if the process is biased or what made it biased. When asserting biases, you must first distinguish them from random noise. Using pg's logic, every selection process that isn't perfect is biased.
azakai 11y ago
>> This is not just a heuristic for detecting bias. It's what bias means. > Under that definition That's not a definition. It's a claim about what the term "bias" means.

suchow 11y ago

This logic is similar to that of Larry Summers in his remarks about diversity in science and engineering, http://www.harvard.edu/president/speeches/summers_2005/nber.php http://www.harvard.edu/president/speeches/summers_2005/nber.....

rewqfdsa 11y ago

Why are you using biased mathematics? If statistics and the scientific method (the tools through which dead white men continue to colonize) give us obviously problematic results, we should abandon them in favor of a method of inquiry that promotes social justice.

WildUtah 11y ago
I enjoyed this comment.
SFLemonade 11y ago
Please tell me this is sarcasm...
- tprice7 11y ago
  Based on rewqfdsa's comment history, this is probably sarcasm.
graycat 11y ago
> social justice Gee, I never saw a definition. Not sure the meaning of the phrase is clear without a definition.

robot22 11y ago

I find it slightly amazing he posted this. Did he not bother to ask someone with a probabilistic background before posting?

yummyfajitas 11y ago

Graham's intuition is assuming equality of the two distributions. As I noted in a different comment here, you can pretty easily fix Graham's test. Compute min(accepted a) and min(accepted B) instead of the means. In your example, the min of the accepted distributions would both work out to be 80%.

tansey 11y ago
So the PG estimator is clearly problematic. I agree that the yummfajitas (YM) estimator looks to be consistent. In this case though, we're dealing with (small) finite sample sizes, so we need to come up with some sort of test statistic. What would the YM test be here? It seems tricky since you are dealing with a conditional distribution based on left-censored data. I'm also not aware of any difference-of-minimums test, though I am happy to be educated if there is one!
- yummyfajitas 11y ago
  I don't know of something to refer to, but I don't think the statistics are too hard. The test statistic would be exactly min(sample1) and min(sample2). Suppose the cutoff sample is distributed according to f(x)H(x-C). Then the probability of the minima of a sample exceeding C+e by random chance, assuming the null hypothesis, is p = (1-\int_C^{C+e}f(x) dx)^N. So now you have a frequentist hypothesis test. If you make reasonable assumptions on f(x) (non-vanishing near C, quantified somehow), it's even nice and non-parametric.
  - tansey 11y ago
    Does that assume both samples are identically distributed and the only difference is the cutoff? If it does, then couldn't we just continue to do a difference of means test and still be consistent? If it doesn't, how do you handle identifying the cutoff minima and the two different distributions in a frequentist way?
    - yummyfajitas 11y ago

ChicagoBoy11 11y ago

Great insight. I had the great fortune to take a course with Gary Becker where we went into some of the mathematics of college admissions. He made this precise point -- the variances of the particular populations you are looking at matter a great detail. He managed to build some pretty convincing models which provided a compelling narrative for "biases" that we seemed to observe in the real world, all with simple changes to the distributions of populations. Great comment.

WildUtah 11y ago
I had the great fortune to take a course with Gary Becker Lucky you. He was great. I never had the chance to take a class from him but it would have been worth a Chicago winter to have the chance.
- ChicagoBoy11 11y ago
  Second-best: https://www.youtube.com/watch?v=QajILZ3S2RE&list=PL9334868E7A821E2A https://www.youtube.com/watch?v=QajILZ3S2RE&list=PL9334868E7... (I took this very class the year prior)

mattmaroon 11y ago

So in this particular case, assuming that First Round's sample size is significant, it may just be that the female founders who seek them out are just on average better than the male ones? I suppose that if women think that the selection process is biased against them, and most do (and it may be) perhaps the less than excellent ones just don't apply, whereas that isn't true for males?

deleted 11y ago
[deleted]
rattray 11y ago
This seems plausible to me (though no less problematic, of course).
Stasis5001 11y ago
The problem is different than that. It's that the measurement of performance is susceptible to differences in the conditional distribution of disparate groups, given that they were selected. We can illustrate it by modifing the GP example to not include time at all, and make the metric perfect. Suppose we are selecting for the next qualifying round for the Olympic 400m team, and select candidates if their 400m is under 50 seconds. Then, we measure performance -- immediately -- by a 400m run. We have two candidate pools: people who compete in the professional circuit, and everybody else. 100% of the former group who apply qualify, while only 10% of the others do. Okay, so now we immediately measure the average 400m time of all professionals, versus average time of all amateurs who can beat 50 seconds. It's pretty reasonable to expect that the professionals might average closer to 45 seconds, while the other group might average around 48 (I'm not a 400m expert, the actual numbers might be off. WR is 43 seconds). According to the article, we now conclude that our selection process is actually biased _against_ professionals! This is at the least very counter-intuitive. Maybe we provide both groups with coaching, and re-test after 6 months, a year, whatever. The professionals will certainly still outperform the amateurs. However, suppose one of the amateurs goes on to greatness, and wins. Wouldn't this obviously be biased against amateurs, according to our intuition?
devalier 11y ago

quietplatypus 11y ago

Very interesting! Seems like I don't understand probability as well as I think I do, since I bought the argument till I ran into this comment. So, what are the effects of variance in different evaluation contexts, and do we have a meaningful way to measure bias if we take variance into account? My initial reactions: - It seems the higher the variance, the better examples you can trot out to say you are not biased against that particular group. since youll always find a member of that group who does amazing. - If distributions of performance are multimodal, its even harder to conclude stuff because different institutions might cut off different modes when selecting the bar. - Modeling the sources of variance may lead to insight into any actual bias.

hibikir 11y ago

The problem here is language and what our actual objectives are. When people complain about bias, they are not really talking about mathematical bias, but about something else: Their idea of fairness. They are talking about discrimination. And when we are discussing that, we can't really think about whether rules are applied fairly or not, but whether the rules produce the outcomes that we want. Let's go for a ludicrous example: We'll accept all applicants whose IQ is higher than their weight in pounds. We'll be explicitly discriminating against heavy people, but at the same time, we have pretty clear implicit biases against men, and ethnic groups who tend to be taller. We might as well have said that we prefer children and Japanese women. There's no need for mathematical bias: The bias comes from the rule selection. So, in your example, if our actual objective is to graduate an even amount of people from groups A and B, we have to, explicitly, make it easier for group B to get the scholarship. And many times organizations have objectives like that. As a more real example, let's consider a police department. If the objective is to have a racial makeup that represents the community, and different races have different drop-out rates, the candidate selection will prefer one kind over the other, precisely to counter the drop-out differential. So when regular people, and not mathematicians, discuss bias, the mathematical definition is unimportant. The one important thing is our stated objectives.

55555 11y ago
Great comment. There are two types of fairness, (a) fair rules, and (b) fair outcome.
- rewqfdsa 11y ago
  Favoring fair outcome over having fair rules is against everything I believe. We should not strive for a participation trophy culture.
  - lexcorvus 11y ago
    Indeed. The problem is that people frequently infer unfair rules from unequal outcomes, without taking into account the possibility of systematic group differences. Alan: I believe in equality of opportunity, not equality of outcome. Bob: How do you know there isn't equality of opportunity? Alan: Well, just look at how unequal the outcomes are! At this point, Bob would be wise to change the subject, because if he pressed on, he might get this: Bob: Can you give me an example? Alan: Group X is underrepresented in Field Y. Bob: Maybe Group X isn't as good at Field Y. Alan: What? That's racist and/or sexist! And if Bob were to make a comment to this effect on Hacker News, he's probably get downvoted. This is because most people agree with Alan, and many of them abuse their downvote privileges to punish ideas they disagree with rather than those that don't further the discussion. This degrades the quality of discourse, but at least it helps reassure the downvoters that they aren't racist and/or sexist.

pg 11y ago

It's true that this test assumes groups of applicants are roughly equal in (distribution of) ability. That is the default assumption in most conversations I've been involved in about bias, and particularly the example I used, but I'll add something making that explicit.

gleb 11y ago
I like the idea, but how do you apply this to power law distribution outcomes and get any statistical significance? I don't know the answer. E.g. the underlying First Round's analysis likely has no statistical significance. Assuming the power law distribution of outcomes top 5 outcomes will account for 97% of value. So we now have a study with n=5. To make the point let's apply this to YC's own portfolio. Assuming Dropbox, AirBnb and Stripe represent 75% of its value, we'll learn that YC is incredibly biased against: * MIT graduates * brother founders * founding teams that do not have female founders * and especially males named Drew Hard to believe these conclusions are correct or actionable
- graycat 11y ago
  > I like the idea, but how do you apply this to power law distribution outcomes and get any statistical significance? I don't know the answer. See my post https://news.ycombinator.com/item?id=10484602 https://news.ycombinator.com/item?id=10484602 where are distribution-free. So "power law", Gaussian, anything else, doesn't matter.
- deleted 11y ago
  [deleted]
compbio 11y ago
If feel the addition: "C" the applicants you're looking at have roughly equal distribution of ability. makes the reasoning more tautological/weak. If we take two dart boards (one for female -, one for male founders) as a visual, where hitting near the bull's eye counts as "startup success". If we take "C" to be true, then the darts would be thrown at random. Now we draw a circle around the bull's eye. Anything landing in this circle we fund. If this circle has a smaller radius on the female dartboard, than on the male dartboard, then evidently the smaller female circle will contain more darts closer to the target (better average performance) than the larger radius male circle. But then we do not even need performance numbers: Smaller radius circles will have less darts in them. Using "C" we only need to know that the male-female accept ratio is not 50%-50% for us to have found a bias. In short: If you see a roughly equal distribution of ability, and (for simplicity) a roughly equal number of female to male fundraisers, then you should always have a roughly equal distribution of female to male founders in your portfolio, performance be damned. The technique is still useful for when you do not have these female vs. male accept ratio's, and a VC publishes only success rates, but this information on ratio's is often more public than success rates/estimates.

vog 11y ago

> But we haven't been biased against A. [...] It was just their prior distribution that was different. This is why that isn't a counter example. Right at the beginning, the article clearly states that this method is only applicable if the prior distribution is equal: | You can use this technique whenever (a) [...], (b) [...], and (c) the groups of applicants you're looking at have roughly equal distribution of ability.

zodiac 11y ago
AFAIK pg added that to the article after WildUtah made his comment, and pg acknowledges this in a reply to WildUtah

jdoliner 11y ago

I think there's a big problem with this counter example. Which is that you're selecting for one thing and judging whether or not you were biased based on another. > Now suppose that we select without bias or inaccuracy all the applicants that have an 80% or better chance of graduation. This is subtly different than selecting for the highest graduation rate possible because it's binary, you want a group with >80% chances not a group with the best chances. Imagine if instead of the distributions you had group A was composed of people with a 100% chance of graduation and B was composed of people with an 80% chance of graduation. Our process does nothing to distinguish between those people because that extra 20% chance of graduation doesn't matter. This brings me to what I think is the fundamental problem with your criticism, it's not clear to me what it means for a group in your example to over perform. If you select a group with the goal of 80% of them graduating it doesn't make sense to call 90% of them graduating an over performance. That only makes sense if your goal up front is to maximize the graduation rate. I think if you rerun your example but instead assume an unbiased strategy that selects for the highest graduation rate possible you'll find that pg's essay makes a lot more sense.

meowface 11y ago

I suppose PG really should have said: >the groups of applicants you're looking at have exactly equal distribution of ability. Rather than "roughly equal". But obviously that makes the whole thing infeasible.

sdegutis 11y ago

> This short comment is not up to pg's usual high standards for his essays. I can almost here him thinking in response, "if I throw a dog a bone, I don't want to know if it tastes good or not."

dalek2point3 11y ago

absolutely spot on. Differences in distribution is only one way in which you could disprove pg. There are others. For example, different "treatment effects". If conditional on getting selected, VCs pay more attention or are more useful for women, then that would be another reason that we would get the pattern pg proposes, but is not due to bias at selection.

learnstats2 11y ago

Your premise discriminates between two groups. It's not clear how you can assign a candidate a 90% chance of graduation. That probability must be a subjective assessment that has come from some (biased) source. In truth, an individual will either graduate or not. In your example, you can assign 0% and 100% probabilities in group A, but you can't in group B. The most plausible mathematical explanation for that is that you collected insufficient relevant information about candidates in group B.