10 ms·

Amazon started using such a system a while back trained on their hiring data, and it started very heavily selecting against applications with female-indicating

by mjhay 4y ago

Amazon started using such a system a while back trained on their hiring data, and it started very heavily selecting against applications with female-indicating proxies, and was abandoned. In another case, After an audit of the algorithm, the resume screening company found that the algorithm found two factors to be most indicative of job performance: their name was Jared, and whether they played high school lacrosse which reads like a comical stereotype of a young male New England WASP; honestly kind of impressive.

jerf 4y ago
In general, if you are a machine learning person and you are using a regression system that "explains" itself somehow, I highly recommend frequently looking at those explanations. Once you're excited that you've built a 90%+ model, crack open the hood of that model before you deploy it and do a sanity check on it. While ML algorithms can occasionally pluck some interesting insights out of the data, in my admittedly-limited experience I've also seen some models put together out of what is clearly just garbage, meaningless weights placed on attributes that can't possibly be that relevant carving out a hyperplane through hyperspace that can't possibly represent reality, despite all the efforts of ML models to hold out validation data and rotate their validation data and everything else to avoid overfitting.
- mjhay 4y ago
  Once these systems get complicated enough, it gets really hard to do that. Unless you are doing linear models or tree ensembles, it can be really difficult to get a clear picture of variable importance/effects. It's one thing to extrapolate locally (fine for linear models), but that has little to do with anything relevant when there's lots of higher-order interactions.
  - RodgerTheGreat 4y ago
    What is a safer assumption? - My model is accurate, for reasons I can't understand, verify, or externally justify - My model is not accurate edit: to be more clear on my stance, if a modeling technique is too difficult to justify or explain, it is not a good technique and should not be used, even if the results seem appealing.
    - mjhay 4y ago
      The second one. It's important to have a clear picture of biases and how your model could fail to generalize. In the case of social data from a deeply biased society, these issues can be seen a mile away.
  - jerf 4y ago
    "Once these systems get complicated enough, it gets really hard to do that." That's why I qualified it. Obviously if you're working in neural nets, you're not going to be cracking open that hood. But one of the dirty secrets of ML is you don't always need those techniques. There's the space of problems where linear regression and similar techniques are all you need, there's the class of problems where it doesn't matter what you throw at it you're not getting anything, and the "in between" where deep learning and such helps certainly has some interesting problems in the space, but it isn't necessarily the majority of problems you encounter in the wild. The scary thing is that not being able to crack open the hood doesn't mean the model isn't complete nonsense. It just means you can't examine it.
tyho 4y ago
The thing is, this is a laughable anecdote because clearly there is no causal relationship between lacrosse and job performance. But should a company care? Surely what matters most is whether a factor is predictive of job performance. We can assume that playing lacrosse was predictive, does it matter to Amazon if the relationship is causal? Obviously this doesn't take into account political pressures to achieve outcomes other than that hiring the candidate most likely to perform well.
- mjhay 4y ago
  Selecting for lacrosse players (which as you say, has no causal relationship to job performance) necessarily selects against non-lacrosse players. Lacrosse players tend to be white and male, Amazon should care, due not only to obvious legal risk, but also because they are excluding candidates who otherwise do as well or better than a particular lacrosse player. Why would it make any kind of sense to artificially shrink their candidate pool? This is also a moral thing, not just a political thing. It's not right to discriminate against people for reasons other than job performance.
  - hbrn 4y ago
    Playing devil's advocate here: > Lacrosse players tend to be white and male I suspect they also tend to attend certain schools, and that could very well be causal. > Why would it make any kind of sense to artificially shrink their candidate pool? It doesn't make sense to shrink it if you have 10 candidates. But if you have thousands of resumes, it's not shrinking, it's filtering. > It's not right to It's naive to expect big corporations to act morally. Best you can do is force them to act legally.
    - eru 4y ago
      > It's naive to expect big corporations to act morally. Best you can do is force them to act legally. That's not the only avenue. Big corporations do what their managers, customers, employees and investors want. Eg restaurants will typically try to serve tasty food, even if there's no legal requirement to do so. Similarly Google has free lunch, because that's what employees want. The recent fad for ESG shows that investor can want 'morality', too, and companies are willing to satisfy them. (I quote 'morality', because different people have different ideas, and often people only want the fig-leaf of morality, but not pay the price for what they would consider real morality. See eg greenwashing.) Customers and investors etc get what they are willing to pay for, and that's not necessarily what they say they want.
boredumb 4y ago
We were only one paper away from the true revelation that female lacrosse players named Jared are by far the ideal new hires. hold on to your papers!
bakugo 4y ago
Ah yes, the classic "we fed a bunch of real world data to an AI, said AI started finding patterns in the data and making decisions based on those patterns, so we shut it down because the patterns weren't politically correct" story. How many times have we heard this one?
- ok_dad 4y ago
  Are you arguing people named Jared with high school lacrosse experience are the pinnacle of human achievement?
  - bakugo 4y ago
    I don't know about the Jared thing but I wouldn't be surprised at all if a high school sport ended up being a legitimate indicator of future job performance, even if not a very significant one.
    - ok_dad 4y ago
      I can agree that high school sports players are probably hard workers, sure. Clearly, hard work is a big part of doing well in life, since life has a lot of hard work (some of it is fun, too). However, you can't just blindly "follow the data" unless you have literally accounted for all of the data that is currently not collected, which we don't do because we cannot collect every metric relevant to this issue.
      - xboxnolifes 4y ago
        > However, you can't just blindly "follow the data" unless you have literally accounted for all of the data that is currently not collected, which we don't do because we cannot collect every metric relevant to this issue. Why not? Isn't that basically what the status quo is for hiring? Why does AI hiring have such a larger bar?
  - fluoridation 4y ago
    The statement verbatim was >the algorithm found two factors to be most indicative of job performance: their name was Jared, and whether they played high school lacrosse That doesn't mean that the "pinnacle of human achievement" necessarily is named Jared and played lacrosse in high school. It means that if you hire people who meet those conditions after filtering using X additional criteria, on average you'll get a more performant workforce than if you hire from the general population after filtering using X additional criteria. The algorithm found an objective truth. Given the data provided and the question asked, the answer should be indisputable. The only reason to reject it should be because someone thinks the answer is incorrect, but that means that either the algorithm is incorrect, the data is incorrect, or the question was poorly phrased.

by mjhay 4y ago

jerf 4y ago

In general, if you are a machine learning person and you are using a regression system that "explains" itself somehow, I highly recommend frequently looking at those explanations. Once you're excited that you've built a 90%+ model, crack open the hood of that model before you deploy it and do a sanity check on it. While ML algorithms can occasionally pluck some interesting insights out of the data, in my admittedly-limited experience I've also seen some models put together out of what is clearly just garbage, meaningless weights placed on attributes that can't possibly be that relevant carving out a hyperplane through hyperspace that can't possibly represent reality, despite all the efforts of ML models to hold out validation data and rotate their validation data and everything else to avoid overfitting.

mjhay 4y ago
Once these systems get complicated enough, it gets really hard to do that. Unless you are doing linear models or tree ensembles, it can be really difficult to get a clear picture of variable importance/effects. It's one thing to extrapolate locally (fine for linear models), but that has little to do with anything relevant when there's lots of higher-order interactions.
- RodgerTheGreat 4y ago
  What is a safer assumption? - My model is accurate, for reasons I can't understand, verify, or externally justify - My model is not accurate edit: to be more clear on my stance, if a modeling technique is too difficult to justify or explain, it is not a good technique and should not be used, even if the results seem appealing.
  - mjhay 4y ago
    The second one. It's important to have a clear picture of biases and how your model could fail to generalize. In the case of social data from a deeply biased society, these issues can be seen a mile away.
- jerf 4y ago
  "Once these systems get complicated enough, it gets really hard to do that." That's why I qualified it. Obviously if you're working in neural nets, you're not going to be cracking open that hood. But one of the dirty secrets of ML is you don't always need those techniques. There's the space of problems where linear regression and similar techniques are all you need, there's the class of problems where it doesn't matter what you throw at it you're not getting anything, and the "in between" where deep learning and such helps certainly has some interesting problems in the space, but it isn't necessarily the majority of problems you encounter in the wild. The scary thing is that not being able to crack open the hood doesn't mean the model isn't complete nonsense. It just means you can't examine it.

tyho 4y ago

The thing is, this is a laughable anecdote because clearly there is no causal relationship between lacrosse and job performance. But should a company care? Surely what matters most is whether a factor is predictive of job performance. We can assume that playing lacrosse was predictive, does it matter to Amazon if the relationship is causal? Obviously this doesn't take into account political pressures to achieve outcomes other than that hiring the candidate most likely to perform well.

mjhay 4y ago
Selecting for lacrosse players (which as you say, has no causal relationship to job performance) necessarily selects against non-lacrosse players. Lacrosse players tend to be white and male, Amazon should care, due not only to obvious legal risk, but also because they are excluding candidates who otherwise do as well or better than a particular lacrosse player. Why would it make any kind of sense to artificially shrink their candidate pool? This is also a moral thing, not just a political thing. It's not right to discriminate against people for reasons other than job performance.
- hbrn 4y ago
  Playing devil's advocate here: > Lacrosse players tend to be white and male I suspect they also tend to attend certain schools, and that could very well be causal. > Why would it make any kind of sense to artificially shrink their candidate pool? It doesn't make sense to shrink it if you have 10 candidates. But if you have thousands of resumes, it's not shrinking, it's filtering. > It's not right to It's naive to expect big corporations to act morally. Best you can do is force them to act legally.
  - eru 4y ago
    > It's naive to expect big corporations to act morally. Best you can do is force them to act legally. That's not the only avenue. Big corporations do what their managers, customers, employees and investors want. Eg restaurants will typically try to serve tasty food, even if there's no legal requirement to do so. Similarly Google has free lunch, because that's what employees want. The recent fad for ESG shows that investor can want 'morality', too, and companies are willing to satisfy them. (I quote 'morality', because different people have different ideas, and often people only want the fig-leaf of morality, but not pay the price for what they would consider real morality. See eg greenwashing.) Customers and investors etc get what they are willing to pay for, and that's not necessarily what they say they want.

boredumb 4y ago

We were only one paper away from the true revelation that female lacrosse players named Jared are by far the ideal new hires. hold on to your papers!

bakugo 4y ago

Ah yes, the classic "we fed a bunch of real world data to an AI, said AI started finding patterns in the data and making decisions based on those patterns, so we shut it down because the patterns weren't politically correct" story. How many times have we heard this one?

ok_dad 4y ago
Are you arguing people named Jared with high school lacrosse experience are the pinnacle of human achievement?
- bakugo 4y ago
  I don't know about the Jared thing but I wouldn't be surprised at all if a high school sport ended up being a legitimate indicator of future job performance, even if not a very significant one.
  - ok_dad 4y ago
    I can agree that high school sports players are probably hard workers, sure. Clearly, hard work is a big part of doing well in life, since life has a lot of hard work (some of it is fun, too). However, you can't just blindly "follow the data" unless you have literally accounted for all of the data that is currently not collected, which we don't do because we cannot collect every metric relevant to this issue.
    - xboxnolifes 4y ago
      > However, you can't just blindly "follow the data" unless you have literally accounted for all of the data that is currently not collected, which we don't do because we cannot collect every metric relevant to this issue. Why not? Isn't that basically what the status quo is for hiring? Why does AI hiring have such a larger bar?
- fluoridation 4y ago
  The statement verbatim was >the algorithm found two factors to be most indicative of job performance: their name was Jared, and whether they played high school lacrosse That doesn't mean that the "pinnacle of human achievement" necessarily is named Jared and played lacrosse in high school. It means that if you hire people who meet those conditions after filtering using X additional criteria, on average you'll get a more performant workforce than if you hire from the general population after filtering using X additional criteria. The algorithm found an objective truth. Given the data provided and the question asked, the answer should be indisputable. The only reason to reject it should be because someone thinks the answer is incorrect, but that means that either the algorithm is incorrect, the data is incorrect, or the question was poorly phrased.