3 ms·

The base rate I'm using isn't the population base rate at all - it's the top 25 college students base rate, sliced by race. Between 2/3 and 5/6 of black colleg

The base rate I'm using isn't the population base rate at all - it's the top 25 college students base rate, sliced by race. Between 2/3 and 5/6 of black college students at the top 25 schools did not meet the usual intelligence/test score cutoffs to get in. Thus, the base rate for "exceeds normal cutoff" for a black student at Princeton is in the range of 1/6-1/3. (In contrast, the base rate for Asian students is likely higher. The base rate for athletes and the children of large donors is likely lower, for much the same reason.) The various factors you discuss might be causes for why most blacks don't exceed the normal cutoff. But that doesn't mean that ignoring it will be an optimal decision for a person attempting to hire high intelligence/test score individuals.

hackinthebochs 10y ago
If you're just going by entrance GPA/test scores, then I would still question the usefulness of that base rate after the person has graduated and also passed whatever screening process your company has. GPA/test scores surely correlate with intelligence, but they can be gamed to a large degree especially by students with the means and motivation (the typical Asian student). As far as I've seen GPA/test scores only correlate with success in the first year or two of college. So again we're left with the question of how meaningful is your base rate at all? I don't think there is any reason to think it provides any information that isn't completely overshadowed by the fact that the person graduated, their college GPA, and their success in your own screening. Using such dubious priors is biasing your decision making based on unjustifiable assumptions, and will likely end up with worse decision making.
- yummyfajitas 10y ago
  See the math here: https://news.ycombinator.com/item?id=11965961 https://news.ycombinator.com/item?id=11965961 You can play with numbers to see exactly how important the base rate is. If you believe your screening process is 99.9999% perfect, then most likely base rates are nearly irrelevant. I don't think my screening process is that good.
  - hackinthebochs 10y ago
    The math isn't what I'm questioning. The math doesn't tell you about the quality of your prior, just how to factor it in. But if your prior is crud, then you end up making worse decisions as a result. By using the prior admissions test scores you are assuming that those factors are relevant to a population that has graduated and passed your own screening. This is not an "uninformed prior", you are explicitly modelling the belief of the relevance of entrance scores. Modelling no belief would be a uniform prior, that is no bias in the population.
    - yummyfajitas 10y ago
      By using the prior admissions test scores you are assuming that those factors are relevant to a population that has graduated and passed your own screening. Unless applying your screening to a population somehow magically eliminates all variation that previously existed across the populations, that's a perfectly valid assumption. The argument that if the prior is wrong, it hurts you is of course true. Similarly a uniform prior in a non-uniform situation will hurt you - see this example for an illustration of how ignoring base rates can spread HIV. https://www.chrisstucchio.com/blog/2016/why_gays_cant_donate_blood.html https://www.chrisstucchio.com/blog/2016/why_gays_cant_donate...
      - hackinthebochs 10y ago
        The difference in the HIV case is that we have an actual population sample that justifies our chosen base rate. That is to say, this isn't a "prior belief", this is a measured statistic. It's applicability is precisely because the population of individuals getting an HIV test is random from the general population, from which we have a measured statistic. But the use of the base rate in this case does not justify the usage of informative priors in every case. >Unless applying your screening to a population somehow magically eliminates all variation that previously existed across the populations But it might. The "population" of graduated + passed screening bears no a priori relationship to the population of entering freshmen precisely because this is a non-random sample, orders of magnitude smaller than the original population. Furthermore, there is further evidence that there is no relationship precisely because of the studies that show that entrance exams bare little correlation to college success past the first year.

hackinthebochs 10y ago

If you're just going by entrance GPA/test scores, then I would still question the usefulness of that base rate after the person has graduated and also passed whatever screening process your company has. GPA/test scores surely correlate with intelligence, but they can be gamed to a large degree especially by students with the means and motivation (the typical Asian student). As far as I've seen GPA/test scores only correlate with success in the first year or two of college. So again we're left with the question of how meaningful is your base rate at all? I don't think there is any reason to think it provides any information that isn't completely overshadowed by the fact that the person graduated, their college GPA, and their success in your own screening. Using such dubious priors is biasing your decision making based on unjustifiable assumptions, and will likely end up with worse decision making.

yummyfajitas 10y ago
See the math here: https://news.ycombinator.com/item?id=11965961 https://news.ycombinator.com/item?id=11965961 You can play with numbers to see exactly how important the base rate is. If you believe your screening process is 99.9999% perfect, then most likely base rates are nearly irrelevant. I don't think my screening process is that good.
- hackinthebochs 10y ago
  The math isn't what I'm questioning. The math doesn't tell you about the quality of your prior, just how to factor it in. But if your prior is crud, then you end up making worse decisions as a result. By using the prior admissions test scores you are assuming that those factors are relevant to a population that has graduated and passed your own screening. This is not an "uninformed prior", you are explicitly modelling the belief of the relevance of entrance scores. Modelling no belief would be a uniform prior, that is no bias in the population.
  - yummyfajitas 10y ago
    By using the prior admissions test scores you are assuming that those factors are relevant to a population that has graduated and passed your own screening. Unless applying your screening to a population somehow magically eliminates all variation that previously existed across the populations, that's a perfectly valid assumption. The argument that if the prior is wrong, it hurts you is of course true. Similarly a uniform prior in a non-uniform situation will hurt you - see this example for an illustration of how ignoring base rates can spread HIV. https://www.chrisstucchio.com/blog/2016/why_gays_cant_donate_blood.html https://www.chrisstucchio.com/blog/2016/why_gays_cant_donate...
    - hackinthebochs 10y ago
      The difference in the HIV case is that we have an actual population sample that justifies our chosen base rate. That is to say, this isn't a "prior belief", this is a measured statistic. It's applicability is precisely because the population of individuals getting an HIV test is random from the general population, from which we have a measured statistic. But the use of the base rate in this case does not justify the usage of informative priors in every case. >Unless applying your screening to a population somehow magically eliminates all variation that previously existed across the populations But it might. The "population" of graduated + passed screening bears no a priori relationship to the population of entering freshmen precisely because this is a non-random sample, orders of magnitude smaller than the original population. Furthermore, there is further evidence that there is no relationship precisely because of the studies that show that entrance exams bare little correlation to college success past the first year.