4 ms·

(a) The document linked by the OP is to a blog post that discusses and extensively quotes another blog post [1] which in turn discusses an actual paper [2]. Nat

by abeppu 12y ago

(a) The document linked by the OP is to a blog post that discusses and extensively quotes another blog post [1] which in turn discusses an actual paper [2]. Naturally the paper is where the good stuff is. (b) Both blog posts somewhat understate the problem. The adversarial examples given in the original paper aren't just classified differently than their parent image -- they're created to receive a specific classification. In the figure 5 of the arxiv version, for example, they show clear images of a school bus, temple, praying mantis, dog, etc, which all received the label "ostrich, Struthio camelus". (c) The blog post at [1] wonders whether humans have similar adversarial inputs. Of course it's possible that we might, but I suspect that we have an easier time than these networks in part because: (i) We often get labeled data on a stream of 'perturbed' related inputs by observing objects in time. If I see a white dog in real life, I don't get just a single image of it. I get a series of overlapping 'images' over a period of time, during which time it may move, I may move, the lighting may change, etc. So in a sense, human experience already includes the some of the perturbations that ML techniques have to introduce manually to become more robust. (ii) We also get to take actions to get more/better perceptual data. If you see something interesting or confusing or just novel, you choose to focus on it, or get a better view because of that interestingness or novelty. The original paper talks about the adversarial examples as being in pockets of low probability. If humans encounter these pockets only rarely, it's because when we see something weird, we want to examine it, after which that particular pocket has higher probability. [1] http://www.i-programmer.info/news/105-artificial-intelligence/7352-the-flaw-lurking-in-every-deep-neural-net.html http://www.i-programmer.info/news/105-artificial-intelligenc... [2] http://arxiv.org/abs/1312.6199 http://arxiv.org/abs/1312.6199 or http://cs.nyu.edu/~zaremba/docs/understanding.pdf http://cs.nyu.edu/~zaremba/docs/understanding.pdf

arjie 12y ago
I'm sure someone else has considered this before, but perhaps optical illusions are similar examples of data that causes pathological behaviour? Many of them constrain our viewpoint or the sequence of images viewed so we'd be unable to take the actions you mention to handle them.
- waps 12y ago
  Exactly. Reading the paper, I have to say I disagree. Human minds are stable, for a very simple reason : they randomize inputs, they implicitly haar-cascade inputs simply because the sensors are inaccurate (they add time-variant noise like any real-world sensor), and they're mounted on a moving platform that constantly shifts position. The observation here should be that brains don't filter out noise, they use it to their advantage, to improve their performance. So it's simple : both problems referred in the text don't exist for noisy data, and it's easy to improve ANN's classifications performance through things akin to adding random noise and haar-cascade like approaches (shifting the input image slightly, in x, y, rotation, white balance, ...), then taking the prediction that you saw most often. You can even make neural nets that do this implicitly (though it's even more expensive). Anecdotally, I do think my own mind has this "problem". There are a large number of things I recognize immediately, but there are also quite a few things I have to look at for a few seconds to even a minute or two (usually geometrical stuff, network plans, or the like) before it "clicks" in my mind and I know what it is. Sometimes that is because I have to wait for the noise level to go down (e.g. exit a tunnel or a building into full sunlight), but usually it's not. I think it's very possible that at such times my mind is simply waiting until the noise in the input kicks it over some decision boundary. Also, I find people often reclassify things after looking at them a little while longer.
- theforgottenone 12y ago
  Yes. But more importantly, we are continuously training our network. There is no terminating "training set" except the set of all considered classifications. Our discovery of ourselves being "wrong" about a thing is our network continually adjusting. We also have the notion of ignorance. These classifiers are often forced to come to a conclusion, instead of having "I don't know, let me look at it from another angle" kind of self-adjustment process. "Aha, it is a cat!" moments do not happen for ai. In us, it would create a whole new layer to wrap the classifier around some uncertainty logic. We would be motivated to examine the reasons behind our initial failure, and use the conclusions to train this new layer, further developing strategies to adapt to those inputs.
  - deleted 12y ago
krick 12y ago
I thought that more or less all of AI researchers admitted now for a long time that NNs aren't human brain and shouldn't reasoned about as if they were, because essentially these are different things. They can be used to study some particular behaviour of the intelligence, to make something more clear about how actual brin wokrs, but one must always remember that they are not actual brain. Recent post by LeCun mentioned that as well. Anyway, that would seem natural to me. Consider numbers on the rightmost image in OP-link (I'll refer to that blogpost as [0]). Program recognises them "correctly". Would you? Well, I do recognize some, but not all of them for sure, and I guess it would be easy to make such dataset so I would lose flat to that NN. That makes almost all of [0] pretty much nonsense to me. Author operates common-language words lacking of technical meaning, like "similar". He point's out that two images similar to human are not similar to the algorithm, but if we keep in mind that similarity is property with respect to some observer (ie human, neural net). And that is natural that what is important for that NN is different from what is important for most of us, humans. After all, it often happens that you don't recognize something on photo, but your friend does, and as far as you know you are both humans. More than that, after he tells you what it is you are like "Oh, right, I see now! Silly me!" So it's kinda common sense that for every two different classifiers you can find two objects that will be classified as similar by one and different by another. Well, yeah, humans have more in common between themselves compared to NNs, no big surprise as well. After all your internal image classifier doesn't receive exact pixel values on input, so of course one can find two images so you won't even see the difference and that NN fails to describe them as similar. So essentially it is saying only that "every particular NN doesn't think like human" which is nothing new. Well, nice catch is that you can easily construct such counter-example manually, but that doesn't seem like "the biggest news in neural networks since the invention of the backpropagation algorithm". More than that, if the first notion (about absense of meaningful features for individual neurons) is completely true, it doesn't explain why deep learning is so successful. So, I'm confused. If it's really something "backpropagation-size discovery" I'm waiting for comment from some expert we all know and trust, who can explain things clearly. You know, Hinton, LeCun. I just don't see what's so important about that paper.
- abeppu 12y ago
  I'm not trying to pretend that the way ANNs work is at all closely tied to how actual brains work. But the tasks we're trying to give to ANNs are generally about seeing some dataset the same way we do. The whole point in training ANNs on data images with human created class labels is to teach them / encode in them a representation of the mental concepts that we apply to the world, which is supposed to be generalizable. We evaluate them by testing their outputs on new inputs for which we have a human-provided label. You say that "similar" is relative to some observer -- but if the trained ANNs differ wildly from humans in judging what is similar, then maybe they're not doing the job as well as we hoped.
Retric 12y ago
People seeing faces in all kinds of things is a vary common mistake which is probably analogous.
- coldtea 12y ago
  Not really. That, if anything, is like overfitting.
Dylan16807 12y ago
My interest is not really the existence of borderline-recognizable inputs, but how you would reach them. It's trivial to make any system classify wrong if you can add arbitrary amounts of noise. But how often can you trick a human by taking a full-frame broad daylight image and perturbing each pixel up to some limit? I expect the number to be pretty low, even if you're depriving them of the ability to perform more examinations of the object.
paulftw 12y ago
If you can slightly distort an image to make NN produce any classification you want that opens many interesting options for steganography. Funniest application would be distorting individual characters of the printed text so that OCR engine and human would "see" two totally different but meaningful messages. There's likely not enough complexity in the OCR NN to do that, but who knows.

by abeppu 12y ago

arjie 12y ago

I'm sure someone else has considered this before, but perhaps optical illusions are similar examples of data that causes pathological behaviour? Many of them constrain our viewpoint or the sequence of images viewed so we'd be unable to take the actions you mention to handle them.

waps 12y ago
Exactly. Reading the paper, I have to say I disagree. Human minds are stable, for a very simple reason : they randomize inputs, they implicitly haar-cascade inputs simply because the sensors are inaccurate (they add time-variant noise like any real-world sensor), and they're mounted on a moving platform that constantly shifts position. The observation here should be that brains don't filter out noise, they use it to their advantage, to improve their performance. So it's simple : both problems referred in the text don't exist for noisy data, and it's easy to improve ANN's classifications performance through things akin to adding random noise and haar-cascade like approaches (shifting the input image slightly, in x, y, rotation, white balance, ...), then taking the prediction that you saw most often. You can even make neural nets that do this implicitly (though it's even more expensive). Anecdotally, I do think my own mind has this "problem". There are a large number of things I recognize immediately, but there are also quite a few things I have to look at for a few seconds to even a minute or two (usually geometrical stuff, network plans, or the like) before it "clicks" in my mind and I know what it is. Sometimes that is because I have to wait for the noise level to go down (e.g. exit a tunnel or a building into full sunlight), but usually it's not. I think it's very possible that at such times my mind is simply waiting until the noise in the input kicks it over some decision boundary. Also, I find people often reclassify things after looking at them a little while longer.
theforgottenone 12y ago
Yes. But more importantly, we are continuously training our network. There is no terminating "training set" except the set of all considered classifications. Our discovery of ourselves being "wrong" about a thing is our network continually adjusting. We also have the notion of ignorance. These classifiers are often forced to come to a conclusion, instead of having "I don't know, let me look at it from another angle" kind of self-adjustment process. "Aha, it is a cat!" moments do not happen for ai. In us, it would create a whole new layer to wrap the classifier around some uncertainty logic. We would be motivated to examine the reasons behind our initial failure, and use the conclusions to train this new layer, further developing strategies to adapt to those inputs.
- deleted 12y ago

krick 12y ago

I thought that more or less all of AI researchers admitted now for a long time that NNs aren't human brain and shouldn't reasoned about as if they were, because essentially these are different things. They can be used to study some particular behaviour of the intelligence, to make something more clear about how actual brin wokrs, but one must always remember that they are not actual brain. Recent post by LeCun mentioned that as well. Anyway, that would seem natural to me. Consider numbers on the rightmost image in OP-link (I'll refer to that blogpost as [0]). Program recognises them "correctly". Would you? Well, I do recognize some, but not all of them for sure, and I guess it would be easy to make such dataset so I would lose flat to that NN. That makes almost all of [0] pretty much nonsense to me. Author operates common-language words lacking of technical meaning, like "similar". He point's out that two images similar to human are not similar to the algorithm, but if we keep in mind that similarity is property with respect to some observer (ie human, neural net). And that is natural that what is important for that NN is different from what is important for most of us, humans. After all, it often happens that you don't recognize something on photo, but your friend does, and as far as you know you are both humans. More than that, after he tells you what it is you are like "Oh, right, I see now! Silly me!" So it's kinda common sense that for every two different classifiers you can find two objects that will be classified as similar by one and different by another. Well, yeah, humans have more in common between themselves compared to NNs, no big surprise as well. After all your internal image classifier doesn't receive exact pixel values on input, so of course one can find two images so you won't even see the difference and that NN fails to describe them as similar. So essentially it is saying only that "every particular NN doesn't think like human" which is nothing new. Well, nice catch is that you can easily construct such counter-example manually, but that doesn't seem like "the biggest news in neural networks since the invention of the backpropagation algorithm". More than that, if the first notion (about absense of meaningful features for individual neurons) is completely true, it doesn't explain why deep learning is so successful. So, I'm confused. If it's really something "backpropagation-size discovery" I'm waiting for comment from some expert we all know and trust, who can explain things clearly. You know, Hinton, LeCun. I just don't see what's so important about that paper.

abeppu 12y ago
I'm not trying to pretend that the way ANNs work is at all closely tied to how actual brains work. But the tasks we're trying to give to ANNs are generally about seeing some dataset the same way we do. The whole point in training ANNs on data images with human created class labels is to teach them / encode in them a representation of the mental concepts that we apply to the world, which is supposed to be generalizable. We evaluate them by testing their outputs on new inputs for which we have a human-provided label. You say that "similar" is relative to some observer -- but if the trained ANNs differ wildly from humans in judging what is similar, then maybe they're not doing the job as well as we hoped.

Retric 12y ago

People seeing faces in all kinds of things is a vary common mistake which is probably analogous.

coldtea 12y ago
Not really. That, if anything, is like overfitting.

Dylan16807 12y ago

My interest is not really the existence of borderline-recognizable inputs, but how you would reach them. It's trivial to make any system classify wrong if you can add arbitrary amounts of noise. But how often can you trick a human by taking a full-frame broad daylight image and perturbing each pixel up to some limit? I expect the number to be pretty low, even if you're depriving them of the ability to perform more examinations of the object.

paulftw 12y ago

If you can slightly distort an image to make NN produce any classification you want that opens many interesting options for steganography. Funniest application would be distorting individual characters of the printed text so that OCR engine and human would "see" two totally different but meaningful messages. There's likely not enough complexity in the OCR NN to do that, but who knows.