7 ms·

> The number of women and men in the data set shouldn't matter (algorithms learn that even if there was 1 woman, if she was hired then it will be positive about

> The number of women and men in the data set shouldn't matter (algorithms learn that even if there was 1 woman, if she was hired then it will be positive about future woman candidates). This is incorrect. The key thing to keep in mind is that they are not just predicting who is a good candidate, they are also ranking by the certainty of their prediction. Lower numbers of female candidates could plausibly lead to lower certainty for the prediction model as it would have less data on those people. I've never trained a model on resumes, but I definitely often see this "lower certainty on minorites" thing for models I do train. The lower certainty would in turn lead to lower rankings for women even without any bias in the data. Now, I'm not saying that Amazon's data isn't biased. I would not be surprised if it were. I'm just saying we should be careful in understanding what is evidence of bias and what is not.

tomp 8y ago
> The lower certainty would in turn lead to lower rankings for women even without any bias in the data. I don't think that's true. "No bias" means that gender is irrelevant (i.e. its correlation with outcome is 0%). Therefore the system shouldn't even take it into account - it would evaluate both men and women just by other criteria (experience, technical skills, etc), and it would have equal amounts of data for both (because it wouldn't even see them as different). You need bias to even separate the dataset into distinct categories.
- theptip 8y ago
  > "No bias" means that gender is irrelevant False. If we're talking about the technical statistical definition, bias means systematic deviation from the underlying truth in the data -- see this article by Chris Stucchio with some images for clarification: https://jacobitemag.com/2017/08/29/a-i-bias-doesnt-mean-what-journalists-want-you-to-think-it-means/ https://jacobitemag.com/2017/08/29/a-i-bias-doesnt-mean-what... "In statistics, a “bias” is defined as a statistical predictor which makes errors that all have the same direction. A separate term — “variance” — is used to describe errors without any particular direction. It’s important to distinguish bias (making errors with a common direction) from variance which is simply inaccuracy with no particular direction."
  - deleted 8y ago
    [deleted]
  - tomp 8y ago
    I think the comments I replied to mean bias as in “sexist bias”.
    - grandmczeb 8y ago
      Bias as in racism, sexism, etc, has multiple definitions, some of which are mutually exclusive.
    - theptip 8y ago
      Well, it was clear that _you_ think so. My point was that you should consider the meaning of the word under which the post you're replying to is correct, especially given that the author was claiming specific domain experience.
      - tomp 8y ago
        The original was: > The lower certainty would in turn lead to lower rankings for women even without any bias in the data. your post said: > If we're talking about the technical statistical definition, bias means systematic deviation from the underlying truth in the data So I think my interpretation is correct, even though it's not "the technically statistically correct usage". You were referring to the bias of the algorithm (i.e. the mean divergence from the mean in the data), whereas we were referring to the "hiring bias" evident in the data. In fact, your "bias" was mentioned as "lower rankings for women" - i.e. "the algorithm would have (statistical) bias even without (sexist) bias in the data" and I was replying that I think that's false.
  - chiefalchemist 8y ago
    Question: So technically, the AI is not bias against women per se, but a set of characteristics / properties, that are more common among women. I'm not trying to split hairs (or argue), as much as further clarify the difference between (the common definition of) human bias and that of statistical bias.
blt 8y ago
It's wrong even if their model doesn't output a certainty (not all classifiers do). Almost all ML algorithms optimize the expected classification error under the training distribution. So if the training data contains 90% men, it's better to classify those men at 100% accuracy and women at 0% accuracy, than it is to classify both with 89.9% accuracy. Any unsophisticated model will do this. gp: "The number of women and men in the data set shouldn't matter (algorithms learn that even if there was 1 woman, if she was hired then it will be positive about future woman candidates)." This is false for typical models.
bcheung 8y ago
How did you control for these things? Wondering what patterns there are that people use to prevent social discrimination. Seems challenging since much of AI, especially classification, is essentially a discrimination algorithm.
- was_boring 8y ago
  There are a few ways you can tackle this issue: 1) have the same algorithm for each group, but train separately (so in the end you have two different weights); 2) over-sample the group under represented in the data; 3) make the penalty more severe for guessing wrongly on female then male applicants during training; 4) apply weights to gender encoding; 5) use more then just resumes as data. This isn't an insurmountable problem, but does require extra work then just "encode, throw it in and see what happens". Amazon only scrapped the original team, but formed a new one in which diversity is a goal for the output.
  - gsich 8y ago
    Or: don't include gender in the training data.
    - kareemsabri 8y ago
      They didn’t. It was discovered through other signals (mention of membership in “women’s” clubs etc.
      - gsich 8y ago
        So they did. It should be obvious that if you don't want to include gender, then you have to sanitize gender-related data.
        s73v3r_ 8y ago
        More than that, though. Graduates of all-women colleges were also caught. If you're using school as a data point, that's extremely hard to sanitize.
screye 8y ago
> The lower certainty would in turn lead to lower rankings for women even without any bias in the data. This is not true. Probabilistic-ly speaking, if we are computing P(hiring | gender); Lower certainty means there is a high variance in prior over women. However, over a large dataset, the "score" would almost certainly be equal to the mean of the distribution, and be independent of the variance. In simpler words, if there was a frequency diagram of scores for each gender (most likely bell curves), then only the peak of the bell curve would matter. The flatness / thinness of the curve would be completely irrelevant to the final score. The peak is the mean, and the flatness is the uncertainty. Only the mean matters.
- titzer 8y ago
  There's not enough information about how their ML algorithm works, nor how large their dataset was for any of the above reasoning to be justified. Fwiw, many ranking functions do indeed take certainty into account, penalizing populations with few data points.
  - ewjordan 8y ago
    If they were using any sort of neural networks approach with stochastic gradient descent, the network would have to spend some "gradient juice" to cut a divot that recognizes and penalizes women's colleges and the like. It wouldn't do this just because there were fewer women in the batches, rather it would just not assign any weight to those factors. Unless they presented lots of unqualified resumes of people not in tech as part of the training, which seems like something someone might think reasonable. Then, the model would (correctly) determine that very few people coming from women's colleges are CS majors, and penalize them. However, I'd still expect a well built model to adjust so that if someone was a CS major, it would adjust accordingly and get rid of any default penalty for being at a particular college. If the whole thing was hand-engineered, then of course all bets are off. It's hard to deal well with unbalanced classes, and as you mentioned, without knowing what their data looks like we can only speculate on what really happened. But I will say this: this is not a general failure of ML, these sorts of problems can be avoided if you know what you're doing, unless your data is garbage.

tomp 8y ago

> The lower certainty would in turn lead to lower rankings for women even without any bias in the data. I don't think that's true. "No bias" means that gender is irrelevant (i.e. its correlation with outcome is 0%). Therefore the system shouldn't even take it into account - it would evaluate both men and women just by other criteria (experience, technical skills, etc), and it would have equal amounts of data for both (because it wouldn't even see them as different). You need bias to even separate the dataset into distinct categories.

theptip 8y ago
> "No bias" means that gender is irrelevant False. If we're talking about the technical statistical definition, bias means systematic deviation from the underlying truth in the data -- see this article by Chris Stucchio with some images for clarification: https://jacobitemag.com/2017/08/29/a-i-bias-doesnt-mean-what-journalists-want-you-to-think-it-means/ https://jacobitemag.com/2017/08/29/a-i-bias-doesnt-mean-what... "In statistics, a “bias” is defined as a statistical predictor which makes errors that all have the same direction. A separate term — “variance” — is used to describe errors without any particular direction. It’s important to distinguish bias (making errors with a common direction) from variance which is simply inaccuracy with no particular direction."
- deleted 8y ago
  [deleted]
- tomp 8y ago
  I think the comments I replied to mean bias as in “sexist bias”.
  - grandmczeb 8y ago
    Bias as in racism, sexism, etc, has multiple definitions, some of which are mutually exclusive.
  - theptip 8y ago
    Well, it was clear that _you_ think so. My point was that you should consider the meaning of the word under which the post you're replying to is correct, especially given that the author was claiming specific domain experience.
    - tomp 8y ago
      The original was: > The lower certainty would in turn lead to lower rankings for women even without any bias in the data. your post said: > If we're talking about the technical statistical definition, bias means systematic deviation from the underlying truth in the data So I think my interpretation is correct, even though it's not "the technically statistically correct usage". You were referring to the bias of the algorithm (i.e. the mean divergence from the mean in the data), whereas we were referring to the "hiring bias" evident in the data. In fact, your "bias" was mentioned as "lower rankings for women" - i.e. "the algorithm would have (statistical) bias even without (sexist) bias in the data" and I was replying that I think that's false.
- chiefalchemist 8y ago
  Question: So technically, the AI is not bias against women per se, but a set of characteristics / properties, that are more common among women. I'm not trying to split hairs (or argue), as much as further clarify the difference between (the common definition of) human bias and that of statistical bias.

blt 8y ago

It's wrong even if their model doesn't output a certainty (not all classifiers do). Almost all ML algorithms optimize the expected classification error under the training distribution. So if the training data contains 90% men, it's better to classify those men at 100% accuracy and women at 0% accuracy, than it is to classify both with 89.9% accuracy. Any unsophisticated model will do this. gp: "The number of women and men in the data set shouldn't matter (algorithms learn that even if there was 1 woman, if she was hired then it will be positive about future woman candidates)." This is false for typical models.

bcheung 8y ago

How did you control for these things? Wondering what patterns there are that people use to prevent social discrimination. Seems challenging since much of AI, especially classification, is essentially a discrimination algorithm.

was_boring 8y ago
There are a few ways you can tackle this issue: 1) have the same algorithm for each group, but train separately (so in the end you have two different weights); 2) over-sample the group under represented in the data; 3) make the penalty more severe for guessing wrongly on female then male applicants during training; 4) apply weights to gender encoding; 5) use more then just resumes as data. This isn't an insurmountable problem, but does require extra work then just "encode, throw it in and see what happens". Amazon only scrapped the original team, but formed a new one in which diversity is a goal for the output.
- gsich 8y ago
  Or: don't include gender in the training data.
  - kareemsabri 8y ago
    They didn’t. It was discovered through other signals (mention of membership in “women’s” clubs etc.
    - gsich 8y ago
      So they did. It should be obvious that if you don't want to include gender, then you have to sanitize gender-related data.
      - s73v3r_ 8y ago
        More than that, though. Graduates of all-women colleges were also caught. If you're using school as a data point, that's extremely hard to sanitize.

screye 8y ago

> The lower certainty would in turn lead to lower rankings for women even without any bias in the data. This is not true. Probabilistic-ly speaking, if we are computing P(hiring | gender); Lower certainty means there is a high variance in prior over women. However, over a large dataset, the "score" would almost certainly be equal to the mean of the distribution, and be independent of the variance. In simpler words, if there was a frequency diagram of scores for each gender (most likely bell curves), then only the peak of the bell curve would matter. The flatness / thinness of the curve would be completely irrelevant to the final score. The peak is the mean, and the flatness is the uncertainty. Only the mean matters.

titzer 8y ago
There's not enough information about how their ML algorithm works, nor how large their dataset was for any of the above reasoning to be justified. Fwiw, many ranking functions do indeed take certainty into account, penalizing populations with few data points.
- ewjordan 8y ago
  If they were using any sort of neural networks approach with stochastic gradient descent, the network would have to spend some "gradient juice" to cut a divot that recognizes and penalizes women's colleges and the like. It wouldn't do this just because there were fewer women in the batches, rather it would just not assign any weight to those factors. Unless they presented lots of unqualified resumes of people not in tech as part of the training, which seems like something someone might think reasonable. Then, the model would (correctly) determine that very few people coming from women's colleges are CS majors, and penalize them. However, I'd still expect a well built model to adjust so that if someone was a CS major, it would adjust accordingly and get rid of any default penalty for being at a particular college. If the whole thing was hand-engineered, then of course all bets are off. It's hard to deal well with unbalanced classes, and as you mentioned, without knowing what their data looks like we can only speculate on what really happened. But I will say this: this is not a general failure of ML, these sorts of problems can be avoided if you know what you're doing, unless your data is garbage.