7 ms·

>The most obvious and immediately concerning place that this issue can be manifested is in human diversity. I swear, when someone starts building autonomous ki

>The most obvious and immediately concerning place that this issue can be manifested is in human diversity. I swear, when someone starts building autonomous killer robots, the first set of concerned articles will probably be asking whether robots were properly trained to target all genders and races with equal accuracy. This is not a sensible way to approach AI ethics. >It was recently reported that Amazon had tried building a machine learning system to screen resumés for recruitment. Since Amazon’s current employee base skews male, the examples of ‘successful hires’ also, mechanistically, skewed male and so, therefore, did this system’s selection of resumés. There is nothing "mechanistic" about this. It depends on how you select sample resumes and how you split them between "good" and "bad" labels. I worked on a similar thing as an "encouraged" side-project at a certain company. Except I realized from day 1 that using AI on resumes is a bad idea and aimed to show this with data. My model was aiming to detect people who will quit or get fired within first 6 month (with the intent of lowering them in priority for interviews, supposedly). It miraculously achieved 85% accuracy... by figuring out how to detect summer interns. Framing this problem as "bias" and especially hyper-focusing everyone's attention on diversity aspect of it is extremely irresponsible. (I'm not saying that's what the author is doing, but that's definitely what's being done at large.) Fundamentally, there are significant higher-level problems with using statistical ML models for things like hiring or crime prediction.

Bartweiss 8y ago
That intern story is excellent; I'm adding it to my bank of "weird AI tricks" like pausing Tetris to avoid losing. More topically, you're quite right to object to that Amazon reference. As far as I can tell, the real story is even worse than mislabeling. Amazon devs wanted a system to spot candidates in resume banks, so they trained it to recognize resumes similar to the ones submitted to Amazon in the past. The entire dataset was 'positive', and output degrees of similarity instead of classifications. Amazon applicants are mostly male while the pool was presumably 50/50, so that was learned as an element of "Amazon-candidate-ness". That's also an interesting story, but from the first publication (in Reuters) it's been framed as an uneven base rate 'inevitably/predictably/mechanistically' producing a biased result. Which is not only untrue but downright backwards, since it implies that the rate in the general data is what matters, rather than the relative rate between samples and positive classifications. It's yet another variant of the mammogram base rates question, and I wish people would stop trying to reinforce the incorrect answer to that.
- john-radio 8y ago
  > That intern story is excellent; I'm adding it to my bank of "weird AI tricks" like pausing Tetris to avoid losing. Post your bank! Let's be like Magnus Carlson and occasionally ask ourselves, "What would DeepMind do?"
  - Bartweiss 8y ago
    Oh man, good question. I'm always up for swapping these stories. A lot of these came from a paper on weird AI tricks, and resulting best-of list on a blog collecting these stories.[1][2] Suffice to say, the people who think the orthogonality thesis is a weird hypothetical aren't keeping up with the state of things. - The aforementioned Tetris story: an undirected learner was set to maximize score at Tetris learned normal play techniques, but also learned to pause the game immediately before losing so that the score wouldn't "decline" at game over. - In the same vein as interns quitting, proxy detection of all sorts. Identify "field with sheep" by finding green fields with grey skies, or letting heuristics like "humans pick up dogs and cats" override correct identifications. (It's a goat until you pick it up, then it's a dog!) - An agent playing Q*bert found a known bug for infinite lives, then escalated to an unknown bug which disabled the game while overflowing the score counter. - Agents in a physics sim tasked with jumping as high as possible instead learned to 'fly' by abusing collision detection bugs, hitting themselves in ways that created upward momentum. - Another "maximize jump height" task demonstrated that "highest" is an extremely fuzzy term. Initially measured by highest point, they became incredible tall. Measured by lowest point, they stayed tall and grew topheavy to 'kick' their base upwards. - Number-handling bugs of all kinds. In one case, small twitches led to floating-point errors that created energy. In another, a "minimize force" task got solved by maximizing force and triggering integer wraparound. My personal favorite is an adversarial bug. An agent playing tic-tac-toe on an infinite grid with a time limit submitted extremely remote moves which caused timeouts/crashes in any agent that tried to model the full board. [1] https://arxiv.org/pdf/1803.03453.pdf https://arxiv.org/pdf/1803.03453.pdf [2] https://aiweirdness.com/post/172894792687/when-algorithms-surprise-us https://aiweirdness.com/post/172894792687/when-algorithms-su...
bduerst 8y ago
If you train an AI using data from a system that already has certain biases, then the AI is going to replicate those same systemic biases in it's own predictions. It follows the "garbage in, garbage out" idiom. Curiously though, did you compare the non-hire (full time) rates of interns vs fire rates of non-interns?
- 6cd6beb 8y ago
  >If you train an AI using data from a system that already has certain biases, then the AI is going to replicate those same systemic biases That's not what happened in the example at all. The example company isn't biased against summer interns, "who stops working after x time" was just a bad question. The comment you're replying to can boil down to "do you want a monkey's paw solving your problem? If so then AI may be for you" Or perhaps "stop pretending you're ever going to get ethics or empathy out of a computer"
  - bduerst 8y ago
    I was referring to the Amazon resume model. The intern hires model was labeling, as GP said.
- gambler 8y ago
  >did you compare the non-hire (full time) rates of interns vs fire rates of non-interns? Not sure I understand the question. IIRC, the way data was setup there was no way to tell why an intern stopped working for the company, because for all interns "reason code" for separation was the same.
jakelazaroff 8y ago
> There is nothing "mechanistic" about this. It depends on how you select sample resumes and how you split them between "good" and "bad" labels. Isn't that what the article is trying to say, though? That your model can only be as accurate as your data set… and that even then, you have to be very careful to make sure it's not inferring patterns from entirely unrelated information?
lonnyk 8y ago
I get the point, but why didn't you just exclude intern resumes from the training data? Do you still suspect a skewed result?
- gambler 8y ago
  >I get the point, but why didn't you just exclude intern resumes from the training data? That was the logical next step and we started on that, but it required exporting more historic data out of the HR system and filtering out anyone who started as an intern as well. Sounds simple, but in practice it's anything but. Just for the reference, data extraction, cleaning and filtering in that project took at least an order of magnitude more time than anything related to machine learning. The project eventually lost steam and got abandoned. >Do you still suspect a skewed result? Absolutely. My personal intuition is that there is very little correlation between resumes and candidate quality. If that is true, any seemingly accurate predictions would be the result of a similar problem. Testing this hypothesis was a large portion of why I agreed to work on the project in the first place.
  - sfreporter 8y ago
    @gambler: thank you for reading my reporting. I would love to chat confidentially to understand your perspective better. Please see my HN profile.
    - johndavis925254 8y ago
      @sfreporter your reporting had the main lede quite buried to create sensationalism. Gender bias is a distant problem if the model's results are completely random. Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs, the people said. With the technology returning results almost at random, Amazon shut down the project, they said.
  - lonnyk 8y ago
    Got it. Thanks for the response.
moomin 8y ago
You’ve just read a long article that covers many aspects and zeroed in on your own hobby horse. You say there’s significantly bigger issues, but you don’t actually talk about that. Instead you talk about the thing you just said you didn’t think people should be talking about. There’s some serious projection going on here.
joshuamorton 8y ago
>Framing this problem as "bias" Except that's exactly what it is. Much as your model was biased against interns. > and especially hyper-focusing everyone's attention on diversity aspect of it is extremely irresponsible. Why? Pointing out a specific and concrete harm badly designed ML models cause is irresponsible? Just because the same kind of methodological flaw can cause other harms its irresponsible to use a motivating example?
- gambler 8y ago
  >Why? Pointing out a specific and concrete harm badly designed ML models cause is irresponsible? In my opinion, yes, if it leads most readers to misjudge some fundamental properties of the problem as a whole. Again, I'm not saying this article is guilty, but most are.
  - joshuamorton 8y ago
    > In my opinion, yes, if it leads most readers to misjudge some fundamental properties of the problem as a whole. Which problem? The general statement of this problem is "models, trained on [somehow] misrepresentative data [or even technically representative data] can draw unintended conclusions that lead to harm". Specifically in this case, the harm was "the model was basically just trained to ignore all women applicants due to bad inference of conditional probabilities". This is a common thing. Because our society draws lines and has bias, its fairly common for modelling failures to exist along those lines. Indeed, sometimes the failures are mostly harmless and immediately obvious, but often they aren't. And people building models should be made aware of those failure scenarios, and be especially aware of failure scenarios that affect underrepresented groups, because those groups are the most likely for the model to fail on if you aren't actively looking for them. And this stuff is pervasive. Facial recognition tech is much worse at noticing the faces of darker skinned people [1]. Some of this is because the people building the common models (eigenfaces etc.) didn't use diverse skin tones, but some of it goes back further, white balance in film was tuned for lighter skin tones until the 90s[2]. Some of that has likely persisted into modern film and camera technology, unfortunately. People working with data need to understand their data. And that means understanding how bias infests their data. > fundamental properties of the problem as a whole You've yet to state the "whole problem" or the fundamental properties that people might misjudge. So I'm unclear what they are. [1]: Arguably an advantage now. [2]: https://petapixel.com/2015/09/19/heres-a-look-at-how-color-film-was-originally-biased-toward-white-people/ https://petapixel.com/2015/09/19/heres-a-look-at-how-color-f...
bumby 8y ago
Isn't this the one of the major concerns of ML, the bias-variance trade-off? By creating a low-variance model, we create a highly biased model that misses some of the important feature relationships necessary to create a truly generalized model? Meaning, isn't it prudent to spending time on this issue?
- BickNowstrom 8y ago
  You are conflating bias (error) with bias (fairness).
  - bumby 8y ago
    Haha my comment was originally drafted to accuse the parent comment of the same. As I read the article, its concerned with error (e.g. misclassification of cancer) but the parent comment translated this to mean the social bias.

Bartweiss 8y ago

That intern story is excellent; I'm adding it to my bank of "weird AI tricks" like pausing Tetris to avoid losing. More topically, you're quite right to object to that Amazon reference. As far as I can tell, the real story is even worse than mislabeling. Amazon devs wanted a system to spot candidates in resume banks, so they trained it to recognize resumes similar to the ones submitted to Amazon in the past. The entire dataset was 'positive', and output degrees of similarity instead of classifications. Amazon applicants are mostly male while the pool was presumably 50/50, so that was learned as an element of "Amazon-candidate-ness". That's also an interesting story, but from the first publication (in Reuters) it's been framed as an uneven base rate 'inevitably/predictably/mechanistically' producing a biased result. Which is not only untrue but downright backwards, since it implies that the rate in the general data is what matters, rather than the relative rate between samples and positive classifications. It's yet another variant of the mammogram base rates question, and I wish people would stop trying to reinforce the incorrect answer to that.

john-radio 8y ago
> That intern story is excellent; I'm adding it to my bank of "weird AI tricks" like pausing Tetris to avoid losing. Post your bank! Let's be like Magnus Carlson and occasionally ask ourselves, "What would DeepMind do?"
- Bartweiss 8y ago
  Oh man, good question. I'm always up for swapping these stories. A lot of these came from a paper on weird AI tricks, and resulting best-of list on a blog collecting these stories.[1][2] Suffice to say, the people who think the orthogonality thesis is a weird hypothetical aren't keeping up with the state of things. - The aforementioned Tetris story: an undirected learner was set to maximize score at Tetris learned normal play techniques, but also learned to pause the game immediately before losing so that the score wouldn't "decline" at game over. - In the same vein as interns quitting, proxy detection of all sorts. Identify "field with sheep" by finding green fields with grey skies, or letting heuristics like "humans pick up dogs and cats" override correct identifications. (It's a goat until you pick it up, then it's a dog!) - An agent playing Q*bert found a known bug for infinite lives, then escalated to an unknown bug which disabled the game while overflowing the score counter. - Agents in a physics sim tasked with jumping as high as possible instead learned to 'fly' by abusing collision detection bugs, hitting themselves in ways that created upward momentum. - Another "maximize jump height" task demonstrated that "highest" is an extremely fuzzy term. Initially measured by highest point, they became incredible tall. Measured by lowest point, they stayed tall and grew topheavy to 'kick' their base upwards. - Number-handling bugs of all kinds. In one case, small twitches led to floating-point errors that created energy. In another, a "minimize force" task got solved by maximizing force and triggering integer wraparound. My personal favorite is an adversarial bug. An agent playing tic-tac-toe on an infinite grid with a time limit submitted extremely remote moves which caused timeouts/crashes in any agent that tried to model the full board. [1] https://arxiv.org/pdf/1803.03453.pdf https://arxiv.org/pdf/1803.03453.pdf [2] https://aiweirdness.com/post/172894792687/when-algorithms-surprise-us https://aiweirdness.com/post/172894792687/when-algorithms-su...

bduerst 8y ago

If you train an AI using data from a system that already has certain biases, then the AI is going to replicate those same systemic biases in it's own predictions. It follows the "garbage in, garbage out" idiom. Curiously though, did you compare the non-hire (full time) rates of interns vs fire rates of non-interns?

6cd6beb 8y ago
>If you train an AI using data from a system that already has certain biases, then the AI is going to replicate those same systemic biases That's not what happened in the example at all. The example company isn't biased against summer interns, "who stops working after x time" was just a bad question. The comment you're replying to can boil down to "do you want a monkey's paw solving your problem? If so then AI may be for you" Or perhaps "stop pretending you're ever going to get ethics or empathy out of a computer"
- bduerst 8y ago
  I was referring to the Amazon resume model. The intern hires model was labeling, as GP said.
gambler 8y ago
>did you compare the non-hire (full time) rates of interns vs fire rates of non-interns? Not sure I understand the question. IIRC, the way data was setup there was no way to tell why an intern stopped working for the company, because for all interns "reason code" for separation was the same.

jakelazaroff 8y ago

> There is nothing "mechanistic" about this. It depends on how you select sample resumes and how you split them between "good" and "bad" labels. Isn't that what the article is trying to say, though? That your model can only be as accurate as your data set… and that even then, you have to be very careful to make sure it's not inferring patterns from entirely unrelated information?

lonnyk 8y ago

I get the point, but why didn't you just exclude intern resumes from the training data? Do you still suspect a skewed result?

gambler 8y ago
>I get the point, but why didn't you just exclude intern resumes from the training data? That was the logical next step and we started on that, but it required exporting more historic data out of the HR system and filtering out anyone who started as an intern as well. Sounds simple, but in practice it's anything but. Just for the reference, data extraction, cleaning and filtering in that project took at least an order of magnitude more time than anything related to machine learning. The project eventually lost steam and got abandoned. >Do you still suspect a skewed result? Absolutely. My personal intuition is that there is very little correlation between resumes and candidate quality. If that is true, any seemingly accurate predictions would be the result of a similar problem. Testing this hypothesis was a large portion of why I agreed to work on the project in the first place.
- sfreporter 8y ago
  @gambler: thank you for reading my reporting. I would love to chat confidentially to understand your perspective better. Please see my HN profile.
  - johndavis925254 8y ago
    @sfreporter your reporting had the main lede quite buried to create sensationalism. Gender bias is a distant problem if the model's results are completely random. Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs, the people said. With the technology returning results almost at random, Amazon shut down the project, they said.
- lonnyk 8y ago
  Got it. Thanks for the response.

moomin 8y ago

You’ve just read a long article that covers many aspects and zeroed in on your own hobby horse. You say there’s significantly bigger issues, but you don’t actually talk about that. Instead you talk about the thing you just said you didn’t think people should be talking about. There’s some serious projection going on here.

joshuamorton 8y ago

>Framing this problem as "bias" Except that's exactly what it is. Much as your model was biased against interns. > and especially hyper-focusing everyone's attention on diversity aspect of it is extremely irresponsible. Why? Pointing out a specific and concrete harm badly designed ML models cause is irresponsible? Just because the same kind of methodological flaw can cause other harms its irresponsible to use a motivating example?

gambler 8y ago
>Why? Pointing out a specific and concrete harm badly designed ML models cause is irresponsible? In my opinion, yes, if it leads most readers to misjudge some fundamental properties of the problem as a whole. Again, I'm not saying this article is guilty, but most are.
- joshuamorton 8y ago
  > In my opinion, yes, if it leads most readers to misjudge some fundamental properties of the problem as a whole. Which problem? The general statement of this problem is "models, trained on [somehow] misrepresentative data [or even technically representative data] can draw unintended conclusions that lead to harm". Specifically in this case, the harm was "the model was basically just trained to ignore all women applicants due to bad inference of conditional probabilities". This is a common thing. Because our society draws lines and has bias, its fairly common for modelling failures to exist along those lines. Indeed, sometimes the failures are mostly harmless and immediately obvious, but often they aren't. And people building models should be made aware of those failure scenarios, and be especially aware of failure scenarios that affect underrepresented groups, because those groups are the most likely for the model to fail on if you aren't actively looking for them. And this stuff is pervasive. Facial recognition tech is much worse at noticing the faces of darker skinned people [1]. Some of this is because the people building the common models (eigenfaces etc.) didn't use diverse skin tones, but some of it goes back further, white balance in film was tuned for lighter skin tones until the 90s[2]. Some of that has likely persisted into modern film and camera technology, unfortunately. People working with data need to understand their data. And that means understanding how bias infests their data. > fundamental properties of the problem as a whole You've yet to state the "whole problem" or the fundamental properties that people might misjudge. So I'm unclear what they are. [1]: Arguably an advantage now. [2]: https://petapixel.com/2015/09/19/heres-a-look-at-how-color-film-was-originally-biased-toward-white-people/ https://petapixel.com/2015/09/19/heres-a-look-at-how-color-f...

bumby 8y ago

Isn't this the one of the major concerns of ML, the bias-variance trade-off? By creating a low-variance model, we create a highly biased model that misses some of the important feature relationships necessary to create a truly generalized model? Meaning, isn't it prudent to spending time on this issue?

BickNowstrom 8y ago
You are conflating bias (error) with bias (fairness).
- bumby 8y ago
  Haha my comment was originally drafted to accuse the parent comment of the same. As I read the article, its concerned with error (e.g. misclassification of cancer) but the parent comment translated this to mean the social bias.