5 ms·

The fact that a human isn't fooled by the attack (we can still recognize the 32x32 images for what they are), points to an interesting gap in the abilities of c

The fact that a human isn't fooled by the attack (we can still recognize the 32x32 images for what they are), points to an interesting gap in the abilities of conventional convolutional neural nets.

bpicolo 9y ago
Not a new concept in general, just a new approach for it with a single pixel change.
deegles 9y ago
That's only because the attack is designed to target that particular network. Just wait until we understand real brains better and can generate tailored attacks...
- lbearl 9y ago
  Isn't that basically what an optical illusion is?
  - alex_anglin 9y ago
    Plus cognitive biases. Those are already well understood and used in a variety of ways.
  - smsm42 9y ago
    Yes, and also this: https://en.wikipedia.org/wiki/Dazzle_camouflage https://en.wikipedia.org/wiki/Dazzle_camouflage Hacks human brain rather efficiently.
    - thaumasiotes 9y ago
      > Dazzle was adopted by the Admiralty in the UK, and then by the United States Navy, with little evaluation. Each ship's dazzle pattern was unique to avoid making classes of ships instantly recognisable to the enemy. The result was that a profusion of dazzle schemes was tried, and the evidence for their success was at best mixed. So many factors were involved that it was impossible to determine which were important, and whether any of the colour schemes were effective.
      - smsm42 9y ago
        It is true that battlefield efficiency of such camouflage is unknown - but I think one can see the effects it does on the brain without conducting a proper rigorous study. The question here is not whether the effect exists - which is IMO obvious - but whether it's enough to make difference in actual combat.
- bno1 9y ago
  500,000 years ago our eyes and vision were probably significantly worse than they are today. As our eyes evolved to capture the world better our brains also evolved to correct the errors from our eyes. On the other hand, we feed into our neural networks high quality images. It's true that they are low resolution but they don't contain noticeable noise or artifacts. The attack described here is a smart application of salt and pepper noise. It's ineffective on humans because our vision evolved to filter it out, but a network which has seen only noiseless images is helpless. I'm curious whether training the network by adding noise and other mutations to the set would make the network more resilient to this attacks. In other words, it's the training set or the network architecture that's vulnerable here?
  - avaxzat 9y ago
    >I'm curious whether training the network by adding noise and other mutations to the set would make the network more resilient to this attacks. In other words, it's the training set or the network architecture that's vulnerable here? This is called adversarial training and is currently the most popular technique for protecting neural networks against this type of attack. That being said, it doesn't work as well as one would hope: the adversarially trained models are usually still vulnerable to other attacks.
- avaxzat 9y ago
  The one-pixel attack works for pretty much any machine learning model, not just one single neural network.
sdenton4 9y ago
A better analogy would be stimulating an individual receptor on the retina, in an eye with only 32^2 such receptors. When we see these pictures, we've got a much larger set of inputs to work with.

The fact that a human isn't fooled by the attack (we can still recognize the 32x32 images for what they are), points to an interesting gap in the abilities of conventional convolutional neural nets.

bpicolo 9y ago

Not a new concept in general, just a new approach for it with a single pixel change.

deegles 9y ago

That's only because the attack is designed to target that particular network. Just wait until we understand real brains better and can generate tailored attacks...

lbearl 9y ago
Isn't that basically what an optical illusion is?
- alex_anglin 9y ago
  Plus cognitive biases. Those are already well understood and used in a variety of ways.
- smsm42 9y ago
  Yes, and also this: https://en.wikipedia.org/wiki/Dazzle_camouflage https://en.wikipedia.org/wiki/Dazzle_camouflage Hacks human brain rather efficiently.
  - thaumasiotes 9y ago
    > Dazzle was adopted by the Admiralty in the UK, and then by the United States Navy, with little evaluation. Each ship's dazzle pattern was unique to avoid making classes of ships instantly recognisable to the enemy. The result was that a profusion of dazzle schemes was tried, and the evidence for their success was at best mixed. So many factors were involved that it was impossible to determine which were important, and whether any of the colour schemes were effective.
    - smsm42 9y ago
      It is true that battlefield efficiency of such camouflage is unknown - but I think one can see the effects it does on the brain without conducting a proper rigorous study. The question here is not whether the effect exists - which is IMO obvious - but whether it's enough to make difference in actual combat.
bno1 9y ago
500,000 years ago our eyes and vision were probably significantly worse than they are today. As our eyes evolved to capture the world better our brains also evolved to correct the errors from our eyes. On the other hand, we feed into our neural networks high quality images. It's true that they are low resolution but they don't contain noticeable noise or artifacts. The attack described here is a smart application of salt and pepper noise. It's ineffective on humans because our vision evolved to filter it out, but a network which has seen only noiseless images is helpless. I'm curious whether training the network by adding noise and other mutations to the set would make the network more resilient to this attacks. In other words, it's the training set or the network architecture that's vulnerable here?
- avaxzat 9y ago
  >I'm curious whether training the network by adding noise and other mutations to the set would make the network more resilient to this attacks. In other words, it's the training set or the network architecture that's vulnerable here? This is called adversarial training and is currently the most popular technique for protecting neural networks against this type of attack. That being said, it doesn't work as well as one would hope: the adversarially trained models are usually still vulnerable to other attacks.
avaxzat 9y ago
The one-pixel attack works for pretty much any machine learning model, not just one single neural network.

sdenton4 9y ago

A better analogy would be stimulating an individual receptor on the retina, in an eye with only 32^2 such receptors. When we see these pictures, we've got a much larger set of inputs to work with.