5 ms·

Does this mean we can stop seeing adversarial examples as a deep fundamental flaw of deep neural networks? Seems like human system experiences them as well!

carlmr 8y ago
We know human systems experience them as well. But we can design roads and other things so that they don't trick us too much. We have very little intuitive design understanding of what can trick robots. And in a lot of cases this is also something how you can hack an AI system if you understand what it misperceives.
- jon_richards 8y ago
  As xkcd points out, you don't have to worry too much about people trying to trick self-driving systems with visual hacks. https://xkcd.com/1958/ https://xkcd.com/1958/
  - carlmr 8y ago
    Ok, fair point about the hacking, but unintentionally misleading things are also hard to estimate.
  - menudo 8y ago
    The XKCD comic strip makes for a useful and shallow zinger, amid casual banter, but the real world actually faces two or three new layers of complexity, which actually reassert the problem it attempts to dispose of. 1. The global reach of networked telecommunications permits a small quantity of sociopathic murderers to operate from beyond jurisdictions that can reach them, and also assists in destroying evidence of their interference. 2. Computational power enables force multiplication, such that even just one sociopathic murderer could exploit software flaws across millions of vehicles, simultaneously. 3. Some software exploits will work against self driving cars, which could never work against an ordinary person, and of course, vice versa, but not so much via remote control at a distance, when people are the operators, while we still lack electronic interfaces to our central nervous system.
    - jon_richards 8y ago
      >visual hacks We were discussing "tricking" the computer vision component of a self-driving car, not getting software access. That's still a concern, but it's an entirely different set of security requirements that we already face in planes and existing cars.
      - menudo 8y ago
        Except XKCD wasn't.
        jon_richards 8y ago
        To me, >painting fake lines on the road, or dropping a cutout of a pedestrian onto a highway sounds like >"tricking" the computer vision component of a self-driving car, not getting software access.
saagarjha 8y ago
Aren't optical illusions just adversarial examples for people?
- aetherson 8y ago
  No, and I wish people would stop that meme. Humans can obviously be tricked, in a variety of ways. But adversarial images take advantage of the fact that image-recognizing neural networks do not fit their image recognition into a full fledged understanding of the world like we do. So a few pixels here and there can make a truck look like a panda and the algorithm never says, "But wait, pandas are mostly black and white and this is mostly yellow," or, "But I don't see legs anywhere, or ears." Optical illusions mostly don't cause high level image misclassifications. To the extent that they are anything similar, they're the reverse: using our general world understanding to cause glitches in our information processing, such as cases where you think something is darker or lighter than it is, or bigger or smaller, or bent or straight. Those are your mind applying rules that are based on "how the world usually appears at a high level" to an image where those rules do not in fact apply.
  - saagarjha 8y ago
    I’m no machine learning expert, but it seems to me that neural networks just don’t really work like people do, as much as people would like to claim that they’ve created something that works like the human brain.
    - aetherson 8y ago
      Sure. So, then, why did you posit a direct parallel between optical illusions and adversarial images?
      - saagarjha 8y ago
        Because that’s another thing I can now mention when people tell me neural networks are like the human brain, since this is an example they often like to bring up.
mannykannot 8y ago
This does not take the issue off the table, as far as practical applications of neural-network object recognition and scene analysis are concerned. Firstly, it leaves open the issue of an additional category of images for which neural networks alone very confidently misidentify things. Secondly, the time limitation, while being a perfectly valid aspect of the experiment, means that parity has not yet been achieved in these cases, either.
tmalsburg2 8y ago
In the experiment, humans showed a minor drop in accuracy (<10%) after seeing the stimulus for 63ms. The presentation was so short that humans made a considerable amount of mistakes even with the non-adversarial examples. In contrast, neural networks get adversarial stimuli consistently wrong even though they are allowed to fully process them. The results also do not prove that the underlying mechanisms are the same in humans and neural networks.
andrewflnr 8y ago
It doesn't decrease the actual set of attacks you can run against a DNN, so no. If anything it increases the scope of the security vulnerability.
andbberger 8y ago
No

carlmr 8y ago

We know human systems experience them as well. But we can design roads and other things so that they don't trick us too much. We have very little intuitive design understanding of what can trick robots. And in a lot of cases this is also something how you can hack an AI system if you understand what it misperceives.

jon_richards 8y ago
As xkcd points out, you don't have to worry too much about people trying to trick self-driving systems with visual hacks. https://xkcd.com/1958/ https://xkcd.com/1958/
- carlmr 8y ago
  Ok, fair point about the hacking, but unintentionally misleading things are also hard to estimate.
- menudo 8y ago
  The XKCD comic strip makes for a useful and shallow zinger, amid casual banter, but the real world actually faces two or three new layers of complexity, which actually reassert the problem it attempts to dispose of. 1. The global reach of networked telecommunications permits a small quantity of sociopathic murderers to operate from beyond jurisdictions that can reach them, and also assists in destroying evidence of their interference. 2. Computational power enables force multiplication, such that even just one sociopathic murderer could exploit software flaws across millions of vehicles, simultaneously. 3. Some software exploits will work against self driving cars, which could never work against an ordinary person, and of course, vice versa, but not so much via remote control at a distance, when people are the operators, while we still lack electronic interfaces to our central nervous system.
  - jon_richards 8y ago
    >visual hacks We were discussing "tricking" the computer vision component of a self-driving car, not getting software access. That's still a concern, but it's an entirely different set of security requirements that we already face in planes and existing cars.
    - menudo 8y ago
      Except XKCD wasn't.
      - jon_richards 8y ago
        To me, >painting fake lines on the road, or dropping a cutout of a pedestrian onto a highway sounds like >"tricking" the computer vision component of a self-driving car, not getting software access.

saagarjha 8y ago

Aren't optical illusions just adversarial examples for people?

aetherson 8y ago
No, and I wish people would stop that meme. Humans can obviously be tricked, in a variety of ways. But adversarial images take advantage of the fact that image-recognizing neural networks do not fit their image recognition into a full fledged understanding of the world like we do. So a few pixels here and there can make a truck look like a panda and the algorithm never says, "But wait, pandas are mostly black and white and this is mostly yellow," or, "But I don't see legs anywhere, or ears." Optical illusions mostly don't cause high level image misclassifications. To the extent that they are anything similar, they're the reverse: using our general world understanding to cause glitches in our information processing, such as cases where you think something is darker or lighter than it is, or bigger or smaller, or bent or straight. Those are your mind applying rules that are based on "how the world usually appears at a high level" to an image where those rules do not in fact apply.
- saagarjha 8y ago
  I’m no machine learning expert, but it seems to me that neural networks just don’t really work like people do, as much as people would like to claim that they’ve created something that works like the human brain.
  - aetherson 8y ago
    Sure. So, then, why did you posit a direct parallel between optical illusions and adversarial images?
    - saagarjha 8y ago
      Because that’s another thing I can now mention when people tell me neural networks are like the human brain, since this is an example they often like to bring up.

mannykannot 8y ago

This does not take the issue off the table, as far as practical applications of neural-network object recognition and scene analysis are concerned. Firstly, it leaves open the issue of an additional category of images for which neural networks alone very confidently misidentify things. Secondly, the time limitation, while being a perfectly valid aspect of the experiment, means that parity has not yet been achieved in these cases, either.

tmalsburg2 8y ago

In the experiment, humans showed a minor drop in accuracy (<10%) after seeing the stimulus for 63ms. The presentation was so short that humans made a considerable amount of mistakes even with the non-adversarial examples. In contrast, neural networks get adversarial stimuli consistently wrong even though they are allowed to fully process them. The results also do not prove that the underlying mechanisms are the same in humans and neural networks.

andrewflnr 8y ago

It doesn't decrease the actual set of attacks you can run against a DNN, so no. If anything it increases the scope of the security vulnerability.

andbberger 8y ago