5 ms·

Don't these kinds of companies have a basic checklist before releasing features like this? And if so, why doesn't this checklist include things like "doesn't id

by daenz 5y ago

Don't these kinds of companies have a basic checklist before releasing features like this? And if so, why doesn't this checklist include things like "doesn't identify members of racial group as animals" ? It makes me think that any woke PR coming from Facebook is lip service, if they don't have even basic checks in place.

saltedonion 5y ago
The space for potential edge cases are so large it will be wack mole. Inference tasks isn’t like software testing where the states are well defined.
- dontreact 5y ago
  Yeah, but it shows some real lack of care to fall into the same edge case that Google was widely criticized for 3 years ago.
  - bryan0 5y ago
    I don’t think this is an edge case. We are asking AI to classify images and videos of people where the results can be disastrous.
- daenz 5y ago
  A large and complex task is well suited for the talented people behind Facebook. But even a list of basic searches that exercised the engine would have caught this. No excuses imo, particularly since it happened to Google[0] not that long ago. Was nobody paying attention? 0. https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai https://www.theverge.com/2018/1/12/16882408/google-racist-go...
  - ta8902 5y ago
    Maybe they thought it would be racist if they explicitly devised rules preventing black people being identified as primates.
- stathibus 5y ago
  Another way to put this is: nobody working in this field has control over what they are building, and can't make any promises about how it will behave. It's like launching a rocket ship that you can't test in a physical simulation first. Something is going to go wrong that can't be predicted because we lack the understanding, but this is seen as an acceptable risk.
  - toxik 5y ago
    Which is inherent to the problem. The domain is the space of _all possible natural images_, it’s so big, it’s ridiculous. Fundamentally, these techniques do not analyze images like humans do, but are rather trained to pick out any salient signal it can latch on to. That seems to be “primates are mostly like humans but darker”, which is superficially true but a pretty weak definition as it includes dark skinned humans.
    - wizzwizz4 5y ago
      It's probably because humans are primates – but the AI systems often have to treat “human” as a completely separate category as “primate”, so they have to draw weird, complex boundaries around “primate” (actually “all non-human primates”). When the “primate” classification is stronger than the “human” classification, the system says “primate” rather than “human”, and if it's predominantly been trained on “pictures of white Americans are not pictures of primates”, its “primate” definition might not be skewed to miss everyone else. I expect you'd get better results if you allowed the system to call humans “primates”, then accept “human primate” as “human” when parsing the output. (That is, leave the “is_primate” output line floating while training on pictures of humans.) I don't know whether that would work, though.
      - toxik 5y ago
        These models typically don’t have hierarchical labels like that, and they apply a softmax to their output - which means /one/ label will be considered correct. (A softmax means taking the exp of your predicted scores, then divide by the sum.)
lgessler 5y ago
Regardless of whether they do, the tech that underlies modern image classification approaches (i.e. deep neural networks) is pathologically hard to understand to the point where it's practically impossible to provide the guarantees you're describing. For example, adding noise to a picture of a panda can make a network think it's a gibbon: https://towardsdatascience.com/breaking-neural-networks-with-adversarial-attacks-f4290a9a45aa https://towardsdatascience.com/breaking-neural-networks-with... That's not to say, of course, that there's nothing to be done. People who aren't white have historically been laughably underrepresented in computer vision datasets, and while that's been changing, I gather that representation still isn't where it needs to be.
dragonwriter 5y ago
> It makes me think that any woke PR coming from Facebook is lip service Corporate PR about anything that isn't concrete things which misleading statements would be both easily revealed and have adverse (e.g., securities-laws-related) consequences is almost 100% lip service, so, yeah, that's a safe assumption, anyway.
ourmandave 5y ago
Did they have even one black person on the AI team? https://www.credera.com/insights/racial-bias-in-machine-learning-and-artificial-intelligence https://www.credera.com/insights/racial-bias-in-machine-lear...
- ratsmack 5y ago
  Why would that have made a difference?
  - ourmandave 5y ago
    Geez, I dunno. If only it was explained in a conveniently linked article.
cfn 5y ago
The question is how would they know they are members of a racial group when it couldn't even identify them as part of the righ species? The answer is that these algorithms are not fit for serious classification and are not appropriate for this type of application. EDIT: pardon me, what I wrote is wrong. We are all primates after all :)

by daenz 5y ago

saltedonion 5y ago

The space for potential edge cases are so large it will be wack mole. Inference tasks isn’t like software testing where the states are well defined.

dontreact 5y ago
Yeah, but it shows some real lack of care to fall into the same edge case that Google was widely criticized for 3 years ago.
- bryan0 5y ago
  I don’t think this is an edge case. We are asking AI to classify images and videos of people where the results can be disastrous.
daenz 5y ago
A large and complex task is well suited for the talented people behind Facebook. But even a list of basic searches that exercised the engine would have caught this. No excuses imo, particularly since it happened to Google[0] not that long ago. Was nobody paying attention? 0. https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai https://www.theverge.com/2018/1/12/16882408/google-racist-go...
- ta8902 5y ago
  Maybe they thought it would be racist if they explicitly devised rules preventing black people being identified as primates.
stathibus 5y ago
Another way to put this is: nobody working in this field has control over what they are building, and can't make any promises about how it will behave. It's like launching a rocket ship that you can't test in a physical simulation first. Something is going to go wrong that can't be predicted because we lack the understanding, but this is seen as an acceptable risk.
- toxik 5y ago
  Which is inherent to the problem. The domain is the space of _all possible natural images_, it’s so big, it’s ridiculous. Fundamentally, these techniques do not analyze images like humans do, but are rather trained to pick out any salient signal it can latch on to. That seems to be “primates are mostly like humans but darker”, which is superficially true but a pretty weak definition as it includes dark skinned humans.
  - wizzwizz4 5y ago
    It's probably because humans are primates – but the AI systems often have to treat “human” as a completely separate category as “primate”, so they have to draw weird, complex boundaries around “primate” (actually “all non-human primates”). When the “primate” classification is stronger than the “human” classification, the system says “primate” rather than “human”, and if it's predominantly been trained on “pictures of white Americans are not pictures of primates”, its “primate” definition might not be skewed to miss everyone else. I expect you'd get better results if you allowed the system to call humans “primates”, then accept “human primate” as “human” when parsing the output. (That is, leave the “is_primate” output line floating while training on pictures of humans.) I don't know whether that would work, though.
    - toxik 5y ago
      These models typically don’t have hierarchical labels like that, and they apply a softmax to their output - which means /one/ label will be considered correct. (A softmax means taking the exp of your predicted scores, then divide by the sum.)

lgessler 5y ago

Regardless of whether they do, the tech that underlies modern image classification approaches (i.e. deep neural networks) is pathologically hard to understand to the point where it's practically impossible to provide the guarantees you're describing. For example, adding noise to a picture of a panda can make a network think it's a gibbon: https://towardsdatascience.com/breaking-neural-networks-with-adversarial-attacks-f4290a9a45aa https://towardsdatascience.com/breaking-neural-networks-with... That's not to say, of course, that there's nothing to be done. People who aren't white have historically been laughably underrepresented in computer vision datasets, and while that's been changing, I gather that representation still isn't where it needs to be.

dragonwriter 5y ago

> It makes me think that any woke PR coming from Facebook is lip service Corporate PR about anything that isn't concrete things which misleading statements would be both easily revealed and have adverse (e.g., securities-laws-related) consequences is almost 100% lip service, so, yeah, that's a safe assumption, anyway.

ourmandave 5y ago

Did they have even one black person on the AI team? https://www.credera.com/insights/racial-bias-in-machine-learning-and-artificial-intelligence https://www.credera.com/insights/racial-bias-in-machine-lear...

ratsmack 5y ago
Why would that have made a difference?
- ourmandave 5y ago
  Geez, I dunno. If only it was explained in a conveniently linked article.

cfn 5y ago

The question is how would they know they are members of a racial group when it couldn't even identify them as part of the righ species? The answer is that these algorithms are not fit for serious classification and are not appropriate for this type of application. EDIT: pardon me, what I wrote is wrong. We are all primates after all :)