3 ms·

There are the practical flaws like "okay, how reliable can this be if it decides that this picture that is clearly a car is instead a camel." But you're right,

There are the practical flaws like "okay, how reliable can this be if it decides that this picture that is clearly a car is instead a camel." But you're right, we can study those empirically and say whether or not they're actually a practical problem. But I think that the more interesting question is, what the hell is actually going on here? What is the neural network actually detecting, that you can change what it is detecting with such innocuous-to-human-eyes changes? Is there indeed some kind of pattern that is genuinely characteristic of camels or whatever that is invisible to us, but "visible" to the neural net? The temptation is to say it's just overtraining, picking up some noise that happens to be the same in various training set photos but is genuinely meaningless in the real world. But the fact that multiple different algorithms have the same or similar adversarial examples, and the fact that deep learning neural networks do appear to perform reasonably well in the real world, argues against that. So if we provisionally think that it's not overtraining, what is it? Could there be a feature of camels that we can't see? Could that tell us something about camels, or vision, or something? And vice-versa, what if our goal isn't a narrow task like "identify the existence of camels," but instead we're trying to get a computer to identify similar features in camels as we are (perhaps because the goal is not simply to identify camels, but to identify camels and then reason about the camel in some way). How do we get a deep learning neural net to not focus on some invisible feature, and instead learn to identify a camel based on features that humans DO perceive?

ivanca 12y ago
Is not about vision, is about emerging patters that arise from noise. If just one piece (deemed important) piece of the pattern differs from the expectation, then the classification fails. Therefore surface blurring does fix the issue, because the number of patterns is reduced (when applied in both the source samples and the user input). Here is a diff of the borders of the pictures (instead of pixels, which is less useful), the one in the left with surface blur and the one in the right without: http://i.imgur.com/3bihGVA.png http://i.imgur.com/3bihGVA.png (The parts marked as completely different are the white ones)
- thisjepisje 12y ago
  What does adding surface blur really fix? Why could you not add similarly subtle network-breaking noise to a blurred image? If you think of blurring as downsampling, it is obvious that you also need downsampled noise to trick the NN.
  - ivanca 12y ago
    The noise is not random, one can see exactly where the artifacts were added (the white parts of the image). And for this solution to work the down sampling must happen inside the program, not in the file system, therefore all network-braking noise would have been already discarded.
    - thisjepisje 12y ago
      And for this solution to work the down sampling must happen inside the program, not in the file system, therefore all network-braking noise would have been already discarded. I see, maybe I just didn't get your point the first time. Anyway, what I was trying to say is that if you do view the downsampling as part of the network/program, you could apply the optimization procedure mentioned in the paper to a network that blurs its input. I assume that this would then generate network-breaking patterns that are imperceptible to the human eye, in the same way as happens in the paper.
- aetherson 12y ago
  Yeah, dude, I know. But why is that piece deemed important? Is it truly important? If it is not truly important, then the algorithm is overtrained -- that is, it has found "patterns" that exist in the training set but only by random chance, and which would not exist in a sample set. But as I mentioned, there seem like there are reasonable reasons to suspect that this is not just overtraining.

ivanca 12y ago

Is not about vision, is about emerging patters that arise from noise. If just one piece (deemed important) piece of the pattern differs from the expectation, then the classification fails. Therefore surface blurring does fix the issue, because the number of patterns is reduced (when applied in both the source samples and the user input). Here is a diff of the borders of the pictures (instead of pixels, which is less useful), the one in the left with surface blur and the one in the right without: http://i.imgur.com/3bihGVA.png http://i.imgur.com/3bihGVA.png (The parts marked as completely different are the white ones)

thisjepisje 12y ago
What does adding surface blur really fix? Why could you not add similarly subtle network-breaking noise to a blurred image? If you think of blurring as downsampling, it is obvious that you also need downsampled noise to trick the NN.
- ivanca 12y ago
  The noise is not random, one can see exactly where the artifacts were added (the white parts of the image). And for this solution to work the down sampling must happen inside the program, not in the file system, therefore all network-braking noise would have been already discarded.
  - thisjepisje 12y ago
    And for this solution to work the down sampling must happen inside the program, not in the file system, therefore all network-braking noise would have been already discarded. I see, maybe I just didn't get your point the first time. Anyway, what I was trying to say is that if you do view the downsampling as part of the network/program, you could apply the optimization procedure mentioned in the paper to a network that blurs its input. I assume that this would then generate network-breaking patterns that are imperceptible to the human eye, in the same way as happens in the paper.
aetherson 12y ago
Yeah, dude, I know. But why is that piece deemed important? Is it truly important? If it is not truly important, then the algorithm is overtrained -- that is, it has found "patterns" that exist in the training set but only by random chance, and which would not exist in a sample set. But as I mentioned, there seem like there are reasonable reasons to suspect that this is not just overtraining.