4 ms·

So, I might be wrong, but my understanding of these attacks is that they require you know what the model of the classifier is. If my understanding is correct,

So, I might be wrong, but my understanding of these attacks is that they require you know what the model of the classifier is. If my understanding is correct, I guess my question is: how general are these attacks? Can we ever say "oh yah, don't try to classify this penguin with your off-the-shelf model, it'll come-up iguana 9/10"

rlanday 9y ago
You don't necessarily need to know the whole model, but you need accurate enough prediction score outputs for each category. I think if you have that, that's enough to numerically approximate the gradient of the output scores and perform gradient descent.
- zeitbeule 9y ago
  Someone just published a new attack at ICLR that does not even need confidence scores but works solely on the final decision output of the network and works as well as the best known attacks (that so far required gradients): https://openreview.net/forum?id=SyZI0GWCZ&noteId=SyZI0GWCZ https://openreview.net/forum?id=SyZI0GWCZ&noteId=SyZI0GWCZ
Donald 9y ago
See https://arxiv.org/abs/1610.08401 https://arxiv.org/abs/1610.08401 for model agnostic examples of adversarial attacks.
- IshKebab 9y ago
  This is much more interesting, thanks!
- jszymborski 9y ago
  This is great stuff! I've noticed that the inter-model universality is OK (a little over 50% for one of the universal pertubations), but that's still pretty good!
- justifier 9y ago
  This is a great paper.. the perturbation map is very reminiscent of psychedelic imagery Psychedelics come up a lot with nn created images, but this is interesting that under the influence of a perturbation suddenly the classifier starts presenting illogical assumptions Also similar to how people recollect their own classifiers unfolding or showing bias when considered under the influence of psychedelics Perhaps there is some deeper analogue with a substance influenced brain's neuronal activity
tsomctl 9y ago
There are different kinds of attacks. This one, and related ones that minutely change the input, require you to know some details about the model. There are other kinds of attacks, where you don't need to know what the specific model is. For example, http://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.html http://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.ht... The fundamental problem is that there are only so many training examples and layers that you can fit in a gpu's memory. The current state of the art neural networks work well for very specific tasks (ie handwriting recognition), but don't generalize.
- nl 9y ago
  Neural networks do generalise well, and their vulnerability to crafted attacks is unrelated to generalisation anyway. Additionally the size of the network isn’t the same thing as GPU memory and you don’t fit training examples in memory anyway (?!).
kumarvvr 9y ago
Is getting to know the model of the classifier so difficult? I mean, for an API, is it not possible to deduce the internal structure of the system based on the Input-output combination?? Just asking.
- mattnewton 9y ago
  If by internal structure gp means the number or structure of the layers, then no, it’s not feasible to deduce this. If they write regular blog posts or publish the architecture in a journal however, and the api gives you probabilities, you could maybe recreate it with a process similar to distillation?
abhishek0318 9y ago
The algorithm mentioned in the paper doesn't need any information about the internal structure of the image classifier.
zeitbeule 9y ago
Someone just published a new attack at ICLR that does not even need confidence scores but works solely on the final decision output of the network and works as well as the best known attacks (that so far required gradients): https://openreview.net/forum?id=SyZI0GWCZ&noteId=SyZI0GWCZ https://openreview.net/forum?id=SyZI0GWCZ&noteId=SyZI0GWCZ

rlanday 9y ago

You don't necessarily need to know the whole model, but you need accurate enough prediction score outputs for each category. I think if you have that, that's enough to numerically approximate the gradient of the output scores and perform gradient descent.

zeitbeule 9y ago
Someone just published a new attack at ICLR that does not even need confidence scores but works solely on the final decision output of the network and works as well as the best known attacks (that so far required gradients): https://openreview.net/forum?id=SyZI0GWCZ&noteId=SyZI0GWCZ https://openreview.net/forum?id=SyZI0GWCZ&noteId=SyZI0GWCZ

Donald 9y ago

See https://arxiv.org/abs/1610.08401 https://arxiv.org/abs/1610.08401 for model agnostic examples of adversarial attacks.

IshKebab 9y ago
This is much more interesting, thanks!
jszymborski 9y ago
This is great stuff! I've noticed that the inter-model universality is OK (a little over 50% for one of the universal pertubations), but that's still pretty good!
justifier 9y ago
This is a great paper.. the perturbation map is very reminiscent of psychedelic imagery Psychedelics come up a lot with nn created images, but this is interesting that under the influence of a perturbation suddenly the classifier starts presenting illogical assumptions Also similar to how people recollect their own classifiers unfolding or showing bias when considered under the influence of psychedelics Perhaps there is some deeper analogue with a substance influenced brain's neuronal activity

tsomctl 9y ago

There are different kinds of attacks. This one, and related ones that minutely change the input, require you to know some details about the model. There are other kinds of attacks, where you don't need to know what the specific model is. For example, http://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.html http://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.ht... The fundamental problem is that there are only so many training examples and layers that you can fit in a gpu's memory. The current state of the art neural networks work well for very specific tasks (ie handwriting recognition), but don't generalize.

nl 9y ago
Neural networks do generalise well, and their vulnerability to crafted attacks is unrelated to generalisation anyway. Additionally the size of the network isn’t the same thing as GPU memory and you don’t fit training examples in memory anyway (?!).

kumarvvr 9y ago

Is getting to know the model of the classifier so difficult? I mean, for an API, is it not possible to deduce the internal structure of the system based on the Input-output combination?? Just asking.

mattnewton 9y ago
If by internal structure gp means the number or structure of the layers, then no, it’s not feasible to deduce this. If they write regular blog posts or publish the architecture in a journal however, and the api gives you probabilities, you could maybe recreate it with a process similar to distillation?

abhishek0318 9y ago

The algorithm mentioned in the paper doesn't need any information about the internal structure of the image classifier.

zeitbeule 9y ago

Someone just published a new attack at ICLR that does not even need confidence scores but works solely on the final decision output of the network and works as well as the best known attacks (that so far required gradients): https://openreview.net/forum?id=SyZI0GWCZ&noteId=SyZI0GWCZ https://openreview.net/forum?id=SyZI0GWCZ&noteId=SyZI0GWCZ