3 ms·

Research is needed in that area, but surely adding some noise / blurring to sample during training and during recognition should help reduce the feasibility of

• shmel 9y ago
  Unfortunately, it is not that simple. There are multiple ways to generate adversarial examples and simple defenses help at most against the simplest attacks. Every now and again a new preprint "one simple trick defeating the latest adversarial defense technique" appears on arxiv. I believe they are linked to the nature of DL (and ML in general) models. We try to capture very tiny manifold of natural images in the space of all possible images. We found a technique that does it well (CNNs). But by the very definition of its training we train it to output some values in a finite number of points. In the same time CNN's output outside of those points are mainly defined by the model's smoothness. We can expect that in a neighborhood of a given image the output will be roughly the same (it allows them to generalize). Far from any point of training set the model can say anything. Adversarial examples basically hint that this smoothness work well only along natural images manifold, once we step outside it is much more chaotic. Or, equivalently, the neighborhood where the CNN gives roughly the same output is a very thin "slice" that closely follows natural images manifold. Why is that? Probably there are some non-trivial topological reasons. But it is exactly what nobody understands now.