5 ms·

I would guess that the training data for the ML set was the set of all resumes and an indicator of whether the candidate was eventually hired (maybe with supple

I would guess that the training data for the ML set was the set of all resumes and an indicator of whether the candidate was eventually hired (maybe with supplemental data about how far in the process the candidate got). Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process?

throwawaymath 8y ago
> Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process? Yes, but only in the obvious sense that we all already knew: tech companies hire more men than women for technology-focused roles. That's not to say it isn't an issue; it is, but it's nothing new, and almost certainly not unique to Amazon. Without significant oversight and manual tuning, any training dataset based wholly or in part on current employees is going to demonstrate a bias against women, because there are strictly fewer women. Moreover it's likely that (for a variety of reasons, both intrinsic and extrinsic) fewer women succeed in the interview process as a ratio compared to the number who actually apply.
- hanselot 8y ago
  > tech companies hire more men than women for technology-focused roles. More men are hired because more men eventually end up taking that branch in the RPG of life. Gender studies is not likely to land you a job at amazon, unless it is to feed bias into algorithms.
- Spooky23 8y ago
  I spent a number of years working in tech organizations where the male:female ratio was 60:40. Now I work in a similar org with the same rules where it's more like 85:15. The difference is that old place transitioned administrative staff to IT roles in the 90s/early 2000s when more things were computerized. Those admins, financial analysts, program analysts were more likely to be female and had degrees in liberal arts, accounting, business/finance, etc. In the newer place, they filtered based on computer-related degree upon hiring. That automatically excludes many women. Once hired, female candidates advance as well or better. Anecdotally, I've hired interns in recent years with no tech-specific qualifications as an experiment. If you select for "smart and gets things done" I don't see much of a disadvantage for many roles. You get some duds too, but it wasn't as dramatic a difference as I expected.
  - hellogoodbyeeee 8y ago
    Do FAANG companies train software developers? I think they are big enough and opinionated enough organizations that they might be better at training a "FAANG quality developer" in a year or so than current University system can in four years. I wish companies were more willing to train willing applicants instead of trying to interview for capabilities.
- mcguire 8y ago
  "...any training dataset based wholly or in part on current employees is going to demonstrate a bias against women,..." "Demonstrate" may be the wrong word. How about "encode". And in the future, "enforce". [Edit] On second thought, it would seem like there would be a way to filter out a "raw proportions bias" (like 80% of the resumes in the data set are male) before training.
elfakyn 8y ago
That gives rise to a very interesting concept: ML-based bias assessments. If you take some real-life hiring data (or other applications such as sentencing or generally human behavior data) and train the AI on it, then run it through a bunch of tests to see whether there's bias, that can reveal trends in the underlying training data. I can't imagine this not already being a thing, but I haven't really heard of people using this method.
- bloak 8y ago
  I don't think you can detect "bias" by running "a bunch of tests". "Bias" is a very slippery concept and is probably essentially subjective. When people say an algorithm is "biased" what they seem to mean is that when the judgements of the algorithm are compared with the judgements of a committee of fair-minded and diligent humans then the number of positive outcomes for members of some fashionable minority that we care about is less that what it was with the human judges. It's hard to automate that. And in any case, if you manipulate the algorithm until it "passes" a test like that then you might not really have improved it: when you turn a measure into a target it ceases to be a good measure.
KevinEldon 8y ago
Maybe, but it seems more likely that the model just didn't work. From the article: "Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs... With the technology returning results almost at random"
- WalterSear 8y ago
  Sounds like they've successfully emulated their real world interview process.
  - sanxiyn 8y ago
    Indeed. I am quite curious about details of the model. For example, the single largest contribution to real world interview process variability is interviewer (for resume screening, who screened that resume, etc.). Wouldn't it be possible to code interviewer as categorical variable and separate resume-intrinsic? effect and interviewer effect? They must have tried this, haven't they?
- throwawaymath 8y ago
  Those aren't mutually exclusive. Technically speaking, the model can return results "almost at random" and still demonstrate a bias against any particular attribute if that bias is evident in the underlying training dataset. If there are strictly fewer women in the underlying training set, the model can still return something resembling a uniform distribution of candidates while exacerbating the diminished representation of women. To give a concrete example: you have a bag of blue dice and red dice. There is a supermajority of blue dice in the bag. Your algorithm selects a single die out of the bag on every iteration. The output sequence of dice numbers appears uniform, but there are more blue dice than red dice in the output sequence.
michaelt 8y ago
Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process? Maybe - but maybe not. Imagine a company with 2 men in HR, 2 women in HR, 40 men in engineering, and 10 women in engineering. That's with gender-blind hiring, reflecting only the 4:1 ratio of male to female CS graduates. If you picked a random male hire, there's a 40/42=95% chance they're an engineer whereas if you picked a random female hire, there's a 10/12=83% chance they're an engineer. Thus if you look over all hires' CVs, due to Bayes’ law the dataset says being male increases the conditional probability you meet engineering hiring requirements - and the ML system picks up on that.
- sanxiyn 8y ago
  Why would one train on "all hires CVs"? It'd be "engineering CVs", moreover it'd be "engineering applicants CVs", not "engineering hires CVs".
  - michaelt 8y ago
    Because you were trying to build a system that would perform CV filtering for your entire company, and you figured Deep Learning would just kinda take care of everything. You're right you'd want to look at applicants' CVs - I skipped over that to make the numbers readily comprehensible.
  - Bartweiss 8y ago
    In the article I noticed: "Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs" That language is a bit ambiguous, it could just mean that the algorithm failed on a wide variety of jobs beyond engineering. But another reading suggests that the algorithm was not asked "is this person a good fit for this role" but instead "what, if anything, is this person qualified for?" If that's the case, then the problem starts to make more sense: the algorithm learned a correlation between male-sounding resumes and being hired for engineering roles. That could produce a biased approach even if the decisions in the training data were gender neutral but position-specific. Of course, it would also mean that an Amazon ML team trained an algorithm with inputs that didn't match to its eventual task, and makes me wonder what they used as a test set... (Anecdotally, Amazon spent quite a while recruiting me for SysEng work I'm wildly unqualified for and uninterested in, even suggesting a switch to applying for that team when I was already in the funnel for something I'm more qualified at. When my resume eventually made it to a syseng engineer, they were rightly baffled that I had landed on in their stack, giving me the sense that something was screwy with how Amazon decides who heads towards which role.)
TangoTrotFox 8y ago
Step into your engineering or computer science department and walk into any upper division class and count the females and count the males. That some companies have upwards of 20% females is more likely indicative of extreme bias in hiring as you're not going to find even remotely close to 20% females there. Enter in most measures of competence and you'll find the division is no different. E.g. - if females were being disproportionately hired because of disproportionately positive performance then this might not be an issue, but there seems to be no evidence for that whatsoever. This view that the only thing holding people back is some sort of social or systemic bias seems to be based on nothing except ideology. Incidentally, it's an ideology I also used to hold. Like a good egalitarian I pushed my wife away from sociology and into majoring in CS. She did perfectly well, as did I. More than a decade later she works with people and I work with code. I've no regrets there, but it's not so clear that my persuasion was really the best idea. Norway is another interesting example here. It is considered by many to be the most gender equal location in the world. Yet you'll still find that nurses are primarily female, doctors are primarily male, and all other 'stereotypical' divisions present in most all developed nations. They tried to change these divisions and with extensive effort were able to effect a roughly constant change in some fields. But again, once that push was relinquished things went just about identically to as they were in very short order. Ultimately we're flexible enough that you can manage to fit a square peg into a round hole at times, but once you stop squeezing that peg goes back to what it wants to be.
- mcguire 8y ago
  "I've no regrets there, but it's not so clear that my persuasion was really the best idea." Does she make more money than a similar person with a degree in sociology?
hellogoodbyeeee 8y ago
Right, so the "AI" is not finding the best candidates, but instead finding candidates that succeed in the old hiring process. What I'd like to see (though I'm sure this would get me laughed out of a manager's office) would be a certain fraction of random hires. Then we could train the model on success statistics based on 60,90,120 day performance (or longer!). At the least, they should interview a certain subset of randomly chosen applicants or else the feedback loop from the interviewing process and the AI is going to grow tighter and tighter.

throwawaymath 8y ago

> Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process? Yes, but only in the obvious sense that we all already knew: tech companies hire more men than women for technology-focused roles. That's not to say it isn't an issue; it is, but it's nothing new, and almost certainly not unique to Amazon. Without significant oversight and manual tuning, any training dataset based wholly or in part on current employees is going to demonstrate a bias against women, because there are strictly fewer women. Moreover it's likely that (for a variety of reasons, both intrinsic and extrinsic) fewer women succeed in the interview process as a ratio compared to the number who actually apply.

hanselot 8y ago
> tech companies hire more men than women for technology-focused roles. More men are hired because more men eventually end up taking that branch in the RPG of life. Gender studies is not likely to land you a job at amazon, unless it is to feed bias into algorithms.
Spooky23 8y ago
I spent a number of years working in tech organizations where the male:female ratio was 60:40. Now I work in a similar org with the same rules where it's more like 85:15. The difference is that old place transitioned administrative staff to IT roles in the 90s/early 2000s when more things were computerized. Those admins, financial analysts, program analysts were more likely to be female and had degrees in liberal arts, accounting, business/finance, etc. In the newer place, they filtered based on computer-related degree upon hiring. That automatically excludes many women. Once hired, female candidates advance as well or better. Anecdotally, I've hired interns in recent years with no tech-specific qualifications as an experiment. If you select for "smart and gets things done" I don't see much of a disadvantage for many roles. You get some duds too, but it wasn't as dramatic a difference as I expected.
- hellogoodbyeeee 8y ago
  Do FAANG companies train software developers? I think they are big enough and opinionated enough organizations that they might be better at training a "FAANG quality developer" in a year or so than current University system can in four years. I wish companies were more willing to train willing applicants instead of trying to interview for capabilities.
mcguire 8y ago
"...any training dataset based wholly or in part on current employees is going to demonstrate a bias against women,..." "Demonstrate" may be the wrong word. How about "encode". And in the future, "enforce". [Edit] On second thought, it would seem like there would be a way to filter out a "raw proportions bias" (like 80% of the resumes in the data set are male) before training.

elfakyn 8y ago

That gives rise to a very interesting concept: ML-based bias assessments. If you take some real-life hiring data (or other applications such as sentencing or generally human behavior data) and train the AI on it, then run it through a bunch of tests to see whether there's bias, that can reveal trends in the underlying training data. I can't imagine this not already being a thing, but I haven't really heard of people using this method.

bloak 8y ago
I don't think you can detect "bias" by running "a bunch of tests". "Bias" is a very slippery concept and is probably essentially subjective. When people say an algorithm is "biased" what they seem to mean is that when the judgements of the algorithm are compared with the judgements of a committee of fair-minded and diligent humans then the number of positive outcomes for members of some fashionable minority that we care about is less that what it was with the human judges. It's hard to automate that. And in any case, if you manipulate the algorithm until it "passes" a test like that then you might not really have improved it: when you turn a measure into a target it ceases to be a good measure.

KevinEldon 8y ago

Maybe, but it seems more likely that the model just didn't work. From the article: "Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs... With the technology returning results almost at random"

WalterSear 8y ago
Sounds like they've successfully emulated their real world interview process.
- sanxiyn 8y ago
  Indeed. I am quite curious about details of the model. For example, the single largest contribution to real world interview process variability is interviewer (for resume screening, who screened that resume, etc.). Wouldn't it be possible to code interviewer as categorical variable and separate resume-intrinsic? effect and interviewer effect? They must have tried this, haven't they?
throwawaymath 8y ago
Those aren't mutually exclusive. Technically speaking, the model can return results "almost at random" and still demonstrate a bias against any particular attribute if that bias is evident in the underlying training dataset. If there are strictly fewer women in the underlying training set, the model can still return something resembling a uniform distribution of candidates while exacerbating the diminished representation of women. To give a concrete example: you have a bag of blue dice and red dice. There is a supermajority of blue dice in the bag. Your algorithm selects a single die out of the bag on every iteration. The output sequence of dice numbers appears uniform, but there are more blue dice than red dice in the output sequence.

michaelt 8y ago

Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process? Maybe - but maybe not. Imagine a company with 2 men in HR, 2 women in HR, 40 men in engineering, and 10 women in engineering. That's with gender-blind hiring, reflecting only the 4:1 ratio of male to female CS graduates. If you picked a random male hire, there's a 40/42=95% chance they're an engineer whereas if you picked a random female hire, there's a 10/12=83% chance they're an engineer. Thus if you look over all hires' CVs, due to Bayes’ law the dataset says being male increases the conditional probability you meet engineering hiring requirements - and the ML system picks up on that.

sanxiyn 8y ago
Why would one train on "all hires CVs"? It'd be "engineering CVs", moreover it'd be "engineering applicants CVs", not "engineering hires CVs".
- michaelt 8y ago
  Because you were trying to build a system that would perform CV filtering for your entire company, and you figured Deep Learning would just kinda take care of everything. You're right you'd want to look at applicants' CVs - I skipped over that to make the numbers readily comprehensible.
- Bartweiss 8y ago
  In the article I noticed: "Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs" That language is a bit ambiguous, it could just mean that the algorithm failed on a wide variety of jobs beyond engineering. But another reading suggests that the algorithm was not asked "is this person a good fit for this role" but instead "what, if anything, is this person qualified for?" If that's the case, then the problem starts to make more sense: the algorithm learned a correlation between male-sounding resumes and being hired for engineering roles. That could produce a biased approach even if the decisions in the training data were gender neutral but position-specific. Of course, it would also mean that an Amazon ML team trained an algorithm with inputs that didn't match to its eventual task, and makes me wonder what they used as a test set... (Anecdotally, Amazon spent quite a while recruiting me for SysEng work I'm wildly unqualified for and uninterested in, even suggesting a switch to applying for that team when I was already in the funnel for something I'm more qualified at. When my resume eventually made it to a syseng engineer, they were rightly baffled that I had landed on in their stack, giving me the sense that something was screwy with how Amazon decides who heads towards which role.)

TangoTrotFox 8y ago

Step into your engineering or computer science department and walk into any upper division class and count the females and count the males. That some companies have upwards of 20% females is more likely indicative of extreme bias in hiring as you're not going to find even remotely close to 20% females there. Enter in most measures of competence and you'll find the division is no different. E.g. - if females were being disproportionately hired because of disproportionately positive performance then this might not be an issue, but there seems to be no evidence for that whatsoever. This view that the only thing holding people back is some sort of social or systemic bias seems to be based on nothing except ideology. Incidentally, it's an ideology I also used to hold. Like a good egalitarian I pushed my wife away from sociology and into majoring in CS. She did perfectly well, as did I. More than a decade later she works with people and I work with code. I've no regrets there, but it's not so clear that my persuasion was really the best idea. Norway is another interesting example here. It is considered by many to be the most gender equal location in the world. Yet you'll still find that nurses are primarily female, doctors are primarily male, and all other 'stereotypical' divisions present in most all developed nations. They tried to change these divisions and with extensive effort were able to effect a roughly constant change in some fields. But again, once that push was relinquished things went just about identically to as they were in very short order. Ultimately we're flexible enough that you can manage to fit a square peg into a round hole at times, but once you stop squeezing that peg goes back to what it wants to be.

mcguire 8y ago
"I've no regrets there, but it's not so clear that my persuasion was really the best idea." Does she make more money than a similar person with a degree in sociology?

hellogoodbyeeee 8y ago

Right, so the "AI" is not finding the best candidates, but instead finding candidates that succeed in the old hiring process. What I'd like to see (though I'm sure this would get me laughed out of a manager's office) would be a certain fraction of random hires. Then we could train the model on success statistics based on 60,90,120 day performance (or longer!). At the least, they should interview a certain subset of randomly chosen applicants or else the feedback loop from the interviewing process and the AI is going to grow tighter and tighter.