3 ms·

They didn’t. It was discovered through other signals (mention of membership in “women’s” clubs etc.

gsich 8y ago
So they did. It should be obvious that if you don't want to include gender, then you have to sanitize gender-related data.
- s73v3r_ 8y ago
  More than that, though. Graduates of all-women colleges were also caught. If you're using school as a data point, that's extremely hard to sanitize.
  - gsich 8y ago
    Then what is the purpose of this? At some point you want this thing to "discriminate" (or "select", if this is a better word) people based on what they have done in life. Which is not negative per se.
    - s73v3r_ 8y ago
      But you don't want it to select based on gender.
      - gsich 8y ago
        Would it though? A school name is essentially just that, no gender information there, even with the "women" prefix. If you discriminate other schools, you can do it too with those. FWIW there could be a difference in performance which the ML finds.
        s73v3r_ 8y ago
        It would. Just because it's not explicitly looking for a "W" in the gender field doesn't mean it's not able to determine gender and discriminate based on that. The article and the discussion is all about how these things, despite not explicitly being told to discriminate based on gender, or race, or any number of factors, can still end up gathering a set of factors that are more prevalent among those groups, and discriminate against those people all the same.
        gsich 8y ago
        >despite not explicitly being told to discriminate based on gender, or race, or any number of factors Then this is completeley useless. You want this "AI" to discriminate based on a number of things. That's the whole point. You want to find people that can work for you. If a specific school or title is a bad indicator (based on what you hired now), then it just is that.
- zaarn 8y ago
  That's not as easy as one might think. Machine learning generally doesn't have any prior opinions about things and will learn any possible correlation in the data. It could for example discover that certain words or sentence structures used in the resume are more likely associated with bad candidates. Later you find out that <protected class> has a huge amount of people that use these certain words/structures while most other people don't. And now the AI discriminates against them. ML will pick up on any possible signal including noise.

So they did. It should be obvious that if you don't want to include gender, then you have to sanitize gender-related data.

s73v3r_ 8y ago
More than that, though. Graduates of all-women colleges were also caught. If you're using school as a data point, that's extremely hard to sanitize.
- gsich 8y ago
  Then what is the purpose of this? At some point you want this thing to "discriminate" (or "select", if this is a better word) people based on what they have done in life. Which is not negative per se.
  - s73v3r_ 8y ago
    But you don't want it to select based on gender.
    - gsich 8y ago
      Would it though? A school name is essentially just that, no gender information there, even with the "women" prefix. If you discriminate other schools, you can do it too with those. FWIW there could be a difference in performance which the ML finds.
      - s73v3r_ 8y ago
        It would. Just because it's not explicitly looking for a "W" in the gender field doesn't mean it's not able to determine gender and discriminate based on that. The article and the discussion is all about how these things, despite not explicitly being told to discriminate based on gender, or race, or any number of factors, can still end up gathering a set of factors that are more prevalent among those groups, and discriminate against those people all the same.
        gsich 8y ago
        >despite not explicitly being told to discriminate based on gender, or race, or any number of factors Then this is completeley useless. You want this "AI" to discriminate based on a number of things. That's the whole point. You want to find people that can work for you. If a specific school or title is a bad indicator (based on what you hired now), then it just is that.
zaarn 8y ago
That's not as easy as one might think. Machine learning generally doesn't have any prior opinions about things and will learn any possible correlation in the data. It could for example discover that certain words or sentence structures used in the resume are more likely associated with bad candidates. Later you find out that <protected class> has a huge amount of people that use these certain words/structures while most other people don't. And now the AI discriminates against them. ML will pick up on any possible signal including noise.