4 ms·

A recruiting tool used by Amazon developed a bias against women despite not being told candidates' genders. It penalized candidates who were graduates of all-wo

A recruiting tool used by Amazon developed a bias against women despite not being told candidates' genders. It penalized candidates who were graduates of all-women's colleges and also those who had the word "women" in their resume (e.g. “women’s chess club captain.”) It had been trained on resumes submitted to Amazon during the previous ten years, so the tool's bias was likely reflective of real human bias in Amazon's recruiting process. 1. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G https://www.reuters.com/article/us-amazon-com-jobs-automatio...

mieseratte 7y ago
I think that’s a great example on needing to sanitize data, but being a member of a women’s club or attending a women’s college is itself telling you the candidates gender so I don’t know that this is deriving protected information from non-protected information.
- jetrink 7y ago
  This is a game of whack-a-mole. Given enough data, the ML system will just find other characteristics or groups of characteristics that act as proxies. It might unfairly penalize candidates in ways that are impossible to detect by human evaluators. The promise of these systems is that if you give them a pile of raw data, they will detect subtle patterns that aid in assessing individuals (be they job applicants, ex-cons, whatever.) However, some of the patterns that they detect are our biases against classes of people. If the solution is to sanitize the data to such a degree that the ML system can no longer infer that someone is a woman or that someone is over 40, etc., then the training data is probably also useless for detecting the non-obvious patterns that we want the system to discover.
  - mieseratte 7y ago
    That it is a “game of whack-a-mole” isn’t clear to me, and you are asserting so without any backing, which is why I asked if anyone has real-world examples. Citing an example where they don’t provide gender but then provide “went to an all-women college” is not an example of proving non-protected information which results in protected-class discrimination, it is an example of telling an ML system protected information in a roundabout way.
    - michaelt 7y ago
      Age is a protected class, and it's extremely difficult to sanitise age from a resume. Let's say the candidate got a BS in EECS in 1987, they've had 6 jobs since graduating, and their first job was cost-optimising floppy disk drives. Do you feed the ML system this data, which is clearly correlated with age? Do you keep the number of jobs but delete the duration, responding to 6-jobs-in-3-years the same as 6-jobs-in-30-years? How do you sanitise the age-correlated data out of job descriptions naming old tech or defunct employers? Do you delete all but the last 5 years of their employment history, assigning no value to experience beyond that?
pkaye 7y ago
Can you give another example?
tareqak 7y ago
From my reading of that article, I think the recruiting tool was fed resumés and a data point saying whether or not the corresponding candidate was hired or not. As a result, the tool not only developed a bias against women, but was effectively evidence that there was bias against women in the original hire / not hire decisions.
- tareqak 7y ago
  I just missed the deadline to edit my post, so I am replying to myself. Looking at the parent comment again, I seemed to have just restated it without adding anything new of my own. I meant to add that my reasoning for why Amazon pulled development of this tool was not just because the tool’s bias, but also because that the existence of the tool and its associated training data could open up Amazon to litigation claiming that their hiring decisions were biased against women in ten year span referred to by the article.
  - AnthonyMouse 7y ago
    It's interesting that nobody even bothered to check whether the bias was illicit. They found something that sounds bad and the immediate response is "OMG bad PR, pull emergency shutdown." They just assume that "women's" is coding for female candidates and not something more specific, like gender-segregated activities that may legitimately produce lower quality candidates than the equivalent integrated activities that exposed the student/candidate to a more diverse cohort population. Probably also doesn't help that some of the biggest gender-segregated institutions are penal in nature, i.e. "reform school for troubled girls" or "women's correctional facility." Did anybody even check whether it also penalizes words like "boys" and "gentlemen's"?
    - tareqak 7y ago
      To be fair, it would probably take more time and money irrespective of litigation to be sure that there was illicit bias than to just delete everything and call the exercise a failure.

mieseratte 7y ago

I think that’s a great example on needing to sanitize data, but being a member of a women’s club or attending a women’s college is itself telling you the candidates gender so I don’t know that this is deriving protected information from non-protected information.

jetrink 7y ago
This is a game of whack-a-mole. Given enough data, the ML system will just find other characteristics or groups of characteristics that act as proxies. It might unfairly penalize candidates in ways that are impossible to detect by human evaluators. The promise of these systems is that if you give them a pile of raw data, they will detect subtle patterns that aid in assessing individuals (be they job applicants, ex-cons, whatever.) However, some of the patterns that they detect are our biases against classes of people. If the solution is to sanitize the data to such a degree that the ML system can no longer infer that someone is a woman or that someone is over 40, etc., then the training data is probably also useless for detecting the non-obvious patterns that we want the system to discover.
- mieseratte 7y ago
  That it is a “game of whack-a-mole” isn’t clear to me, and you are asserting so without any backing, which is why I asked if anyone has real-world examples. Citing an example where they don’t provide gender but then provide “went to an all-women college” is not an example of proving non-protected information which results in protected-class discrimination, it is an example of telling an ML system protected information in a roundabout way.
  - michaelt 7y ago
    Age is a protected class, and it's extremely difficult to sanitise age from a resume. Let's say the candidate got a BS in EECS in 1987, they've had 6 jobs since graduating, and their first job was cost-optimising floppy disk drives. Do you feed the ML system this data, which is clearly correlated with age? Do you keep the number of jobs but delete the duration, responding to 6-jobs-in-3-years the same as 6-jobs-in-30-years? How do you sanitise the age-correlated data out of job descriptions naming old tech or defunct employers? Do you delete all but the last 5 years of their employment history, assigning no value to experience beyond that?

pkaye 7y ago

Can you give another example?

tareqak 7y ago

From my reading of that article, I think the recruiting tool was fed resumés and a data point saying whether or not the corresponding candidate was hired or not. As a result, the tool not only developed a bias against women, but was effectively evidence that there was bias against women in the original hire / not hire decisions.

tareqak 7y ago
I just missed the deadline to edit my post, so I am replying to myself. Looking at the parent comment again, I seemed to have just restated it without adding anything new of my own. I meant to add that my reasoning for why Amazon pulled development of this tool was not just because the tool’s bias, but also because that the existence of the tool and its associated training data could open up Amazon to litigation claiming that their hiring decisions were biased against women in ten year span referred to by the article.
- AnthonyMouse 7y ago
  It's interesting that nobody even bothered to check whether the bias was illicit. They found something that sounds bad and the immediate response is "OMG bad PR, pull emergency shutdown." They just assume that "women's" is coding for female candidates and not something more specific, like gender-segregated activities that may legitimately produce lower quality candidates than the equivalent integrated activities that exposed the student/candidate to a more diverse cohort population. Probably also doesn't help that some of the biggest gender-segregated institutions are penal in nature, i.e. "reform school for troubled girls" or "women's correctional facility." Did anybody even check whether it also penalizes words like "boys" and "gentlemen's"?
  - tareqak 7y ago
    To be fair, it would probably take more time and money irrespective of litigation to be sure that there was illicit bias than to just delete everything and call the exercise a failure.