5 ms·
As a defensive measure, why can't random noise just be added to the image prior to classification attempt?
by oh_sigh 10y ago
As a defensive measure, why can't random noise just be added to the image prior to classification attempt?
- lottin 10y agoHow would that help?
- spott 10y agoBecause all the adversarial images that are out there look like noise. If you are training each image with a different amount of noise on top of it, then plausibly you could train the classifier to be insensitive to noise like this overlaid on an image.
- scythe 10y agoI think you might be able to define the vector space "NoiseSpace" in which the random noise is generated as a subspace of the space of all images "ImageSpace", and generate an adverserial example using the quotient space ImageSpace / NoiseSpace.
- antognini 10y agoThe issue is essentially due to the curse of dimensionality. Since your input is very high dimensional, you can find some direction where you don't have go very far to maximize some other output. You can find a few of these adversarial examples and add them to the training set, but there are going to be an exponential number of other directions where you only have to go a little farther. So you can't just add all these perturbations to your training set because there are too many of them. It's a hard problem. (I've been thinking about this issue a lot for my research...)
- m-j-fox 10y agoI think she means you can add white noise to the front end making the output a little bit non-deterministic. Since it won't give the same confidence number twice, it frustrates gradient descent.
- antognini 10y agoJust adding white noise won't work. If you add white noise to an image the NN will make almost exactly the same prediction. The issue with adversarial images is that you add a very particular perturbation to the image, not just any perturbation at all.
- darawk 10y agoI think the idea proposed is that if you add white noise on top of the adversarial perturbation, it will destroy that very particular perturbation.
- esoterica 10y agoIt won't destroy the perturbation. There is a low dimensional manifold along which the cost function will decrease/increase. The adversarial perturbation lies on that manifold, but random white noise (which is a random high dimensional vector) will have close to zero length with high probability when projected onto the manifold and hence won't affect the cost function.
- antognini 10y agoOhhhh, I see. I'm not sure whether that would work. I would have to actually do the experiment. It may not work because this bizarro region of parameter space may be somewhat robust to perturbations, so in a sense you may have to travel out same way you travelled in. But, then again, maybe not. Edit: I should add that these perturbations appear to be very robust to different architectures and datasets. So, the same adversarial perturbation will trick different NNs that were trained on different datasets. This suggests that it will probably be fairly robust to noise. But maybe not! I'm not aware of this experiment having been done.
- AlexCoventry 10y agoIf the attacker knows the distribution of your random noise, they can factor that in to the adversarial search process. From the article: "Adversarial examples are hard to defend against because it is difficult to construct a theoretical model of the adversarial example crafting process. Adversarial examples are solutions to an optimization problem that is non-linear and non-convex for many ML models, including neural networks. Because we don’t have good theoretical tools for describing the solutions to these complicated optimization problems, it is very hard to make any kind of theoretical argument that a defense will rule out a set of adversarial examples."