8 ms·

Of course having data that contradicts the patterns that the neural network is looking for is going to make it err even when is subtle. Humans have it easier be

by ivanca 12y ago

Of course having data that contradicts the patterns that the neural network is looking for is going to make it err even when is subtle. Humans have it easier because we handle more abstract concepts. One way to solve this is "simplifying" the data: In practical terms (for images) that means applying a bilateral filter[0], also know in Photoshop as "surface blur". [0] http://en.m.wikipedia.org/wiki/Bilateral_filter http://en.m.wikipedia.org/wiki/Bilateral_filter

streptomycin 12y ago
That's not the interesting part, the interesting part is: What is even more shocking is that the adversarial examples seem to have some sort of universality. That is a large fraction were misclassified by different network architectures trained on the same data and by networks trained on a different data set.
- ivanca 12y ago
  Yeah, It is the interesting part. Because with sufficiently large inputs (even from different sources) the networks are going to be looking for the same patterns.
  - streptomycin 12y ago
    But why would different structures of networks fit to different data result in exactly the same type of "overfitting"?
darkmighty 12y ago
Are you sure this really negates the problem? (doesn't that essentially make it equivalent to a downsampled input?) I think a more important question than trying to come up with a work around is to consider whether those cases really matter. If the probability of those "adversarial noises" is low enough, who cares? Those cases becomes a curiosity. We have noisy systems that may fail catastrophically operating everywhere with some low probability noise vector, but we manage those probabilities. Unless of course it is an adversarial application and the network is made public; but then I think there are other also simple ways of defeating the attack, e.g. some random smoothing (so that the smoothing isn't also attacked), adding some more noise, etc.
- aetherson 12y ago
  There are the practical flaws like "okay, how reliable can this be if it decides that this picture that is clearly a car is instead a camel." But you're right, we can study those empirically and say whether or not they're actually a practical problem. But I think that the more interesting question is, what the hell is actually going on here? What is the neural network actually detecting, that you can change what it is detecting with such innocuous-to-human-eyes changes? Is there indeed some kind of pattern that is genuinely characteristic of camels or whatever that is invisible to us, but "visible" to the neural net? The temptation is to say it's just overtraining, picking up some noise that happens to be the same in various training set photos but is genuinely meaningless in the real world. But the fact that multiple different algorithms have the same or similar adversarial examples, and the fact that deep learning neural networks do appear to perform reasonably well in the real world, argues against that. So if we provisionally think that it's not overtraining, what is it? Could there be a feature of camels that we can't see? Could that tell us something about camels, or vision, or something? And vice-versa, what if our goal isn't a narrow task like "identify the existence of camels," but instead we're trying to get a computer to identify similar features in camels as we are (perhaps because the goal is not simply to identify camels, but to identify camels and then reason about the camel in some way). How do we get a deep learning neural net to not focus on some invisible feature, and instead learn to identify a camel based on features that humans DO perceive?
  - ivanca 12y ago
    Is not about vision, is about emerging patters that arise from noise. If just one piece (deemed important) piece of the pattern differs from the expectation, then the classification fails. Therefore surface blurring does fix the issue, because the number of patterns is reduced (when applied in both the source samples and the user input). Here is a diff of the borders of the pictures (instead of pixels, which is less useful), the one in the left with surface blur and the one in the right without: http://i.imgur.com/3bihGVA.png http://i.imgur.com/3bihGVA.png (The parts marked as completely different are the white ones)
    - 12y ago

by ivanca 12y ago

streptomycin 12y ago

That's not the interesting part, the interesting part is: What is even more shocking is that the adversarial examples seem to have some sort of universality. That is a large fraction were misclassified by different network architectures trained on the same data and by networks trained on a different data set.

ivanca 12y ago
Yeah, It is the interesting part. Because with sufficiently large inputs (even from different sources) the networks are going to be looking for the same patterns.
- streptomycin 12y ago
  But why would different structures of networks fit to different data result in exactly the same type of "overfitting"?

darkmighty 12y ago

Are you sure this really negates the problem? (doesn't that essentially make it equivalent to a downsampled input?) I think a more important question than trying to come up with a work around is to consider whether those cases really matter. If the probability of those "adversarial noises" is low enough, who cares? Those cases becomes a curiosity. We have noisy systems that may fail catastrophically operating everywhere with some low probability noise vector, but we manage those probabilities. Unless of course it is an adversarial application and the network is made public; but then I think there are other also simple ways of defeating the attack, e.g. some random smoothing (so that the smoothing isn't also attacked), adding some more noise, etc.

aetherson 12y ago
There are the practical flaws like "okay, how reliable can this be if it decides that this picture that is clearly a car is instead a camel." But you're right, we can study those empirically and say whether or not they're actually a practical problem. But I think that the more interesting question is, what the hell is actually going on here? What is the neural network actually detecting, that you can change what it is detecting with such innocuous-to-human-eyes changes? Is there indeed some kind of pattern that is genuinely characteristic of camels or whatever that is invisible to us, but "visible" to the neural net? The temptation is to say it's just overtraining, picking up some noise that happens to be the same in various training set photos but is genuinely meaningless in the real world. But the fact that multiple different algorithms have the same or similar adversarial examples, and the fact that deep learning neural networks do appear to perform reasonably well in the real world, argues against that. So if we provisionally think that it's not overtraining, what is it? Could there be a feature of camels that we can't see? Could that tell us something about camels, or vision, or something? And vice-versa, what if our goal isn't a narrow task like "identify the existence of camels," but instead we're trying to get a computer to identify similar features in camels as we are (perhaps because the goal is not simply to identify camels, but to identify camels and then reason about the camel in some way). How do we get a deep learning neural net to not focus on some invisible feature, and instead learn to identify a camel based on features that humans DO perceive?
- ivanca 12y ago
  Is not about vision, is about emerging patters that arise from noise. If just one piece (deemed important) piece of the pattern differs from the expectation, then the classification fails. Therefore surface blurring does fix the issue, because the number of patterns is reduced (when applied in both the source samples and the user input). Here is a diff of the borders of the pictures (instead of pixels, which is less useful), the one in the left with surface blur and the one in the right without: http://i.imgur.com/3bihGVA.png http://i.imgur.com/3bihGVA.png (The parts marked as completely different are the white ones)
  - 12y ago