4 ms·

As a potential real world example: I'm still not entirely convinced that Google's early models (as used in Images and Photos), and their infamous inability to t

As a potential real world example: I'm still not entirely convinced that Google's early models (as used in Images and Photos), and their infamous inability to tell black people apart from gorillas, was entirely an accidental occurrence. Clearly, such an association would not have been the company's intent, and a properly-produced model would not have presented it. However, a bad actor could have used one of these methods to taint the output. It's unclear the extent of the damage this incident caused, but it serves as a lesson in the unexpected vectors one's business can be attacked from, given the nature of this technology.

whoami_nr 3y ago
Author here. I get what you mean and I remember the incident happening when I was in college. However, I also remember that they were reproduced across multiple publications which means you are implying some sort of data poisoning attack which were super nascent back then. IIRC the spam filter data poisoning was the first class of these vulnerabilities and the image classifier stuff came later. Could be wrong on the timelines. Funnily, they fixed by just removed the gorilla label from their classifier.
- underlipton 3y ago
  >However, I also remember that they were reproduced across multiple publications which means you are implying some sort of data poisoning attack which were super nascent back then. Essentially. I am in no way technical, but my suspicion had been that it was something not even Google was aware could be possible or so effective; by the time they'd caught on, it would have been impossible to reverse without rebuilding the entire thing, having been embedded deeply in the model. The attack being unheard of at the time would then be why it was successful at all. The alternative is simple oversight, which admittedly would be characteristic of Google's regard for DEI and AI safety. Part of me wants it to be a purposeful rogue move because that alternative kind of sucks more. >Funnily, they fixed by just removed the gorilla label from their classifier. I'd heard this, though I think it's more unfortunate than funny. There are a lot of other common terms that you can't search for in Google Photos, in particular, and I wouldn't be surprised to find that they were removed because of similarly unfortunate associations. It severely limits search usability.
ShamelessC 3y ago
This speculation is entirely baseless.
- deleted 3y ago
  [deleted]
- underlipton 3y ago
  Okay.
  - deleted 3y ago
    [deleted]
  - ShamelessC 3y ago
    That’s rude.
    - underlipton 3y ago
      I could say the same.
      - ShamelessC 3y ago
        Okay.
        underlipton 3y ago
        ...That's rude.
        ShamelessC 3y ago
        I could say the same.
AnthonyMouse 3y ago
> I'm still not entirely convinced that Google's early models (as used in Images and Photos), and their infamous inability to tell black people apart from gorillas, was entirely an accidental occurrence. Clearly, such an association would not have been the company's intent, and a properly-produced model would not have presented it. Nah, you're just misunderstanding the kind of bias that produced it. Humans and gorillas are both primates. They look quite similar to each other and certainly more similar to each other than either of them are to a lizard or a bowling ball or a tree. It's the kind of mistake you'd expect it to be likely to make in general. Now suppose you have a training data set with both humans and gorillas, with a human racial composition that reflects the makeup of the population of the first world, i.e. the majority have light skin. You ask it to classify a human with light skin, it's not completely sure if it's a human or a gorilla, but most of the humans it was trained on have light skin and basically all of the gorillas have dark skin, so it skews human. You show it a human with dark skin, it skews gorilla. Models make that kind of error all the time -- it might usually be able to guess right for a billiard ball but its guess for the black 8-ball is a bowling ball. But some errors have political significance because if a human did that we would assume they meant something by it, and then people want to ascribe that intent to the model.
- underlipton 3y ago
  This is a rationalization that fairly easy to debunk (in two ways, even): 1) Gorillas do not have dark skin the way that humans have dark skin. In fact, many features differentiate gorillas from all humans, including dark-skinned ones. The bias present in the model has to overcome these differences to categorize dark-skinned humans with gorillas rather than light-skinned humans. It's reasonable to question how this could happen accidentally. 2) The contention is that the data was poisoned. How, exactly, do you tell apart a case where not enough black people were included in the training data on purpose, versus as a result of institutional racism? In fact, does it matter? One could hold that the data poisoning simply happened at a different part of the process. Also, this: >with a human racial composition that reflects the makeup of the population of the first world, i.e. the majority have light skin. is arguable. The "first world" (within the US/Western political sphere of influence) includes billions of people who would be considered brown/black, including black, indigenous, and mestizo South and North Americans, black Africans, inhabitants of select West Asian countries, East Asians in Japan, and indigenous inhabitants of Oceania. What you mean to say is the "a human racial composition that reflects the makeup of populations Google engineers deem important," which is again damning in its own way.

whoami_nr 3y ago

Author here. I get what you mean and I remember the incident happening when I was in college. However, I also remember that they were reproduced across multiple publications which means you are implying some sort of data poisoning attack which were super nascent back then. IIRC the spam filter data poisoning was the first class of these vulnerabilities and the image classifier stuff came later. Could be wrong on the timelines. Funnily, they fixed by just removed the gorilla label from their classifier.

underlipton 3y ago
>However, I also remember that they were reproduced across multiple publications which means you are implying some sort of data poisoning attack which were super nascent back then. Essentially. I am in no way technical, but my suspicion had been that it was something not even Google was aware could be possible or so effective; by the time they'd caught on, it would have been impossible to reverse without rebuilding the entire thing, having been embedded deeply in the model. The attack being unheard of at the time would then be why it was successful at all. The alternative is simple oversight, which admittedly would be characteristic of Google's regard for DEI and AI safety. Part of me wants it to be a purposeful rogue move because that alternative kind of sucks more. >Funnily, they fixed by just removed the gorilla label from their classifier. I'd heard this, though I think it's more unfortunate than funny. There are a lot of other common terms that you can't search for in Google Photos, in particular, and I wouldn't be surprised to find that they were removed because of similarly unfortunate associations. It severely limits search usability.

ShamelessC 3y ago

This speculation is entirely baseless.

deleted 3y ago
[deleted]
underlipton 3y ago
Okay.
- deleted 3y ago
  [deleted]
- ShamelessC 3y ago
  That’s rude.
  - underlipton 3y ago
    I could say the same.
    - ShamelessC 3y ago
      Okay.
      - underlipton 3y ago
        ...That's rude.
        ShamelessC 3y ago
        I could say the same.

AnthonyMouse 3y ago

> I'm still not entirely convinced that Google's early models (as used in Images and Photos), and their infamous inability to tell black people apart from gorillas, was entirely an accidental occurrence. Clearly, such an association would not have been the company's intent, and a properly-produced model would not have presented it. Nah, you're just misunderstanding the kind of bias that produced it. Humans and gorillas are both primates. They look quite similar to each other and certainly more similar to each other than either of them are to a lizard or a bowling ball or a tree. It's the kind of mistake you'd expect it to be likely to make in general. Now suppose you have a training data set with both humans and gorillas, with a human racial composition that reflects the makeup of the population of the first world, i.e. the majority have light skin. You ask it to classify a human with light skin, it's not completely sure if it's a human or a gorilla, but most of the humans it was trained on have light skin and basically all of the gorillas have dark skin, so it skews human. You show it a human with dark skin, it skews gorilla. Models make that kind of error all the time -- it might usually be able to guess right for a billiard ball but its guess for the black 8-ball is a bowling ball. But some errors have political significance because if a human did that we would assume they meant something by it, and then people want to ascribe that intent to the model.

underlipton 3y ago
This is a rationalization that fairly easy to debunk (in two ways, even): 1) Gorillas do not have dark skin the way that humans have dark skin. In fact, many features differentiate gorillas from all humans, including dark-skinned ones. The bias present in the model has to overcome these differences to categorize dark-skinned humans with gorillas rather than light-skinned humans. It's reasonable to question how this could happen accidentally. 2) The contention is that the data was poisoned. How, exactly, do you tell apart a case where not enough black people were included in the training data on purpose, versus as a result of institutional racism? In fact, does it matter? One could hold that the data poisoning simply happened at a different part of the process. Also, this: >with a human racial composition that reflects the makeup of the population of the first world, i.e. the majority have light skin. is arguable. The "first world" (within the US/Western political sphere of influence) includes billions of people who would be considered brown/black, including black, indigenous, and mestizo South and North Americans, black Africans, inhabitants of select West Asian countries, East Asians in Japan, and indigenous inhabitants of Oceania. What you mean to say is the "a human racial composition that reflects the makeup of populations Google engineers deem important," which is again damning in its own way.