7 ms·

Does Deep Learning Have Deep Flaws?

110 points by vkhuc 12y ago

fleitz 12y ago
But then we wouldn't see faces in clouds... Neutral networks are not perfect solutions, they are solutions that get an organism to reproduce successfully. Read any book on color vision humans have similar problems, yet for the most part we see things, and realize that clouds are just clouds and not faces, except for the religious, they lose their shit when faces appear in clouds.
jrochkind1 12y ago
First thought: Can I turn all digital pictures of me into 'adversarial examples', so the eye of sauron can't identify me from pictures? I'm sure it's not as simple as that, presumably any algorithmic modification to an 'adversarial' nature can be countered by other algorithms. But I predict a new realm of 'arms race' here in the future.
- cpa 12y ago
  Adversarial examples are tied to a specific algorithm, they didn't produce any universal adversarial examples from what I understand.
  - tormeh 12y ago
    If I understood it correctly, they looked at the activation of the individual high-level neurons and edited the image such that the high-level activation values changed a lot while the image was still identical to the human eye.
  - streptomycin 12y ago
    From the article: What’s more surprising is that the same perturbation can cause a different network, which was trained on a different training dataset, to misclassify the same image. It means that adversarial examples are somewhat universal.
    - cpa 12y ago
      Indeed, but they change the dataset and keep the same algorithm. I didn't see anything about changing the algorithm in itself. That's not very surprising: if you have enough training data, any large training set will be typical because it'll be average enough. But when you choose a specific algorithm, you can exploit its weaknesses and throwing more data won't change anything. Change the algorithm will, though.
      - streptomycin 12y ago
        But why would different training data lead to the same error? I could imagine it would lead to something with the same type of flaw, but why do the same exact adversarial images work, out of the near infinite number of possible images? Doesn't intuitively make sense to me, but I can't say I have much of a background in machine learning. Like if you fit 5 close-to-linear 2d points with a 4th order polynomial, you'll overfit. Change the data slightly and you'll still overfit, but your fit will be very different.
nikhizzle 12y ago
I wonder if subtle make up could be used to make a real life face into an adversarial example.
- tormeh 12y ago
  You can apply facial makeup that makes your face look completely different to a human observer. Trying to understand how a person with full-coverage facial makeup really looks like is a very frustrating experience. You're trying to see the contours of the face, but your brain keeps focusing on the applied colors. The same makeup would probably work well on a computer as well.
- admsyn 12y ago
  Yes, actually http://cvdazzle.com/ http://cvdazzle.com/
agibsonccc 12y ago
I think just like any machine learning algorithm, especially with computer vision, you need to prepare things properly. More robust data and images such as moving window on image patches and image rotations, even going as far as applying filters like sobel for edge detection will make your algorithm a lot better. Any algorithms have weaknesses. It's a matter of being aware of them and compensating for them in your model, possibly by using another model.
- JabavuAdams 12y ago
  The value proposition of deep learning is to eliminate these kind of hand-coded features and to discover the features automagically. However, maybe there's a middle ground. I.e. maybe we don't need the more esoteric features that SIFT uses, but it just makes sense to do edge detection, and say a Fourier transform for audio.
  - Punoxysm 12y ago
    The true value proposition of deep learning is not to avoid hand-coded features, but to make better use of scale in data and computational resources. More specifically, adding SIFT or edge detection to your raw pixel input will almost always strictly improve a deep model's performance (though they might be redundant) at a not-particularly-large computational cost. It wouldn't solve the adversarial example problem though, except to the extent that it makes calculating gradients harder.
  - joe_the_user 12y ago
    The value proposition of deep learning is to eliminate these kind of hand-coded features and to discover the features automagically. Yes and just to clarify - this seems like an indication that however useful, deep learning can't follow through with that promise. And I think the ability to do this the key thing - all the approaches work at some level but without this "automagicity", each become hostage to brow-wrinkling experts who become the only one to understand the black-magic of algorithm tuning.
  - agibsonccc 12y ago
    I wrote [1], I'm plenty aware of the "feature discovery" that goes on, but it can still be an enhancer. See my recent talk[2] for a good overall idea of the situation. Normalization and other data transforms are still required for discovery of features. [1]: http://deeplearning4j.org/ http://deeplearning4j.org/ [2]: https://www.youtube.com/watch?v=hykoKDl1AtE https://www.youtube.com/watch?v=hykoKDl1AtE
kilbasa 12y ago
IIRC even the human brain has the 'adversarial' image flaw (these images will be unique to each person), but one simple workaround is to alter the input image via eye movement (which happens unconsciously).
- codeflo 12y ago
  IIRC? Can you provide any source or example? This sounds very interesting.
  - kilbasa 12y ago
    There was a discussion on slashdot (take it with a grain of salt perhaps) about a similar article. http://slashdot.org/story/14/05/27/1326219/the-flaw-lurking-in-every-deep-neural-net http://slashdot.org/story/14/05/27/1326219/the-flaw-lurking-... The comment I recalled was written by someone with the handle "presidenteloco".
jal278 12y ago
The key claim, from the original paper: > Second, we find that deep neural networks learn input-output mappings that are fairly discontinuous [...] Specifically, we find that we can cause the network to misclassify an image by applying a certain imperceptible perturbation [...] the same perturbation can cause a different network that was trained on a different subset of the dataset, to misclassify the same input. It's an interesting outcome -- but there are many deep-learning approaches and many different benchmarks, so it will be important to see if this is a misleading anecdote or indicative of a systematic problem. [1] http://cs.nyu.edu/~zaremba/docs/understanding.pdf http://cs.nyu.edu/~zaremba/docs/understanding.pdf
varelse 12y ago
This problem was observed 20+ years ago with linear models used for protein structure prediction. For any given model of what described a properly folded protein, one could locate conformations of the same protein that were rated as folded even better than the correct conformation (I called them doppelgangers, but the name "decoy" is what caught on). The statistical naivete of the field led to all sorts of inadvertent mixing of training and test set data which generated a lot of spurious claims for solving the problem. That is until one attempted to find those decoys and they were always found. This led to the creation of the CASP competition to weed this out and the field finally moved forward. http://en.wikipedia.org/wiki/CASP http://en.wikipedia.org/wiki/CASP The key similarity to what I described above is that adversarial search is done posterior to the training of the deep neural network. That makes all the difference in the world IMO. These adversaries may just be strange, otherwise hard to reach bad neighborhoods in image space without using a roadmap. Or they may be an unvaoidable consequence of the curse of dimensionality. http://en.wikipedia.org/wiki/Curse_of_dimensionality http://en.wikipedia.org/wiki/Curse_of_dimensionality But given that neural networks have a gradient, it doesn't shock me that it can serve as a roadmap to locate a set of correlated but seemingly minor changes to an example in order to flip its classification. Doing so is simply back-propagation with constant weight values to propagate the gradient to the input data itself - literally a couple lines of code. IMO there are two interesting experiments to do next (not that anyone will take this seriously I expect, but ya know, hear me now, believe me later): 1. Characterize the statistical nature of the changes in input images and then use those summary statistics as the basis of an image altering algorithm to see if that can be used to flip the classification of any image on its own. If it can, be afraid, your driverless car may have blind spots. If not, then this is probably just a narrower form of overfitting. 2. If it's likely overfitting, attempt an expectation maximization-like fix to the problem. Train the network. Generate adversaries, Add them to the training set, train again and then lather rinse repeat until either the network can't be trained or the problem goes away. Expensive? Yes. But you're Google/Facebook/Microsoft and you have lots of GPUs. No excuses... Failing that, the above is on my todo list so I'm throwing it out there to see if anyone can poke holes in the approach.
- magicalist 12y ago
  > If it's likely overfitting, attempt an expectation maximization-like fix to the problem. Train the network. Generate adversaries, Add them to the training set, train again and then lather rinse repeat until either the network can't be trained or the problem goes away. As I quoted in my other comment, the paper suggests doing exactly that.
iandanforth 12y ago
Let's not forget that the word "imperceptible" is a heavily laden term in this context. There are numerous modifications to the data that would be "imperceptible" to a machine learning system, but would completely confuse a human. For example if you were to divide the image into a grid, and shuffle the squares, many ML systems would be tolerant to this kind of modification because some training regimes do this anyway. To that system you haven't changed anything important about the image and it would correctly classify it. What this result says to me is that there are really useful features of the data that can successfully classify images that humans are totally unaware of! And that's neat.
- segmondy 12y ago
  I disagree with your opinion. What this says to me is that DNN is not how humans classifies images.
  - maaku 12y ago
    Which would be defeating a strawman. I don't know anyone who claims DNN is precisely how any aspect of the human brain works.
- galvan 12y ago
  In the linked paper [0], they actually tested that point: after applying Gaussian noise to samples, the model could still recognize them half the time[1], despite being nearly unreadable (to me). [0] http://cs.nyu.edu/~zaremba/docs/understanding.pdf http://cs.nyu.edu/~zaremba/docs/understanding.pdf [1] http://puu.sh/9B4eG/1e9f7eb56b.png http://puu.sh/9B4eG/1e9f7eb56b.png
hyperion2010 12y ago
Not an example from deep learning, but [1] also demonstrates that Bayesian systems also have similar problems with sensitivity to initial conditions that are quite similar. It is also rather striking that these DLNs seem to be tricked by what we would typically think of as noise. 1. http://arxiv.org/abs/1308.6306 http://arxiv.org/abs/1308.6306
- hyperbovine 12y ago
  It is not so striking when you consider how the typical published "result" in deep learning is obtained -- spend a few months turning all the different knobs that these models have to offer (while possibly inventing a few new ones) until its performance meets or beats the current state of the art on MNIST, CIFAR, and related benchmarks. Which is to say, these models heavily overfit to a few datasets; it should not come as a big surprise when they generalize poorly. What happens now will be that this perturbed dataset will be added to the standard training corpus and the DLNs will become robust to this effect. Then someone will figure out new way to mess them up, and the process will repeat.
  - anonymousDan 12y ago
    Would you classify this as progress?
  - wantrepeneur 12y ago
    I don't know what papers you are reading, but you seem to have a very distorted view of the literature. How do results on image net, production speech recognition datasets, language modeling, and high resolution satellite images fit the pattern you allege? Once again HN comment quality is depressingly low when it comes to machine learning topics. But I guess to be fair the description in the original link is very confused as well and misunderstands the conclusions of the paper.
- apw 12y ago
  I thought the Owhadi et al. paper was about model mispecification; i.e., the true model is not in the hypothesis space. That's pretty fundamentally different--and far less of a problem--than gradient descent's "sensitivity to initial conditions".
magicalist 12y ago
The concept is very cool, but it's not surprising that dimensionality reduction through a non-linear process is going to result in sections of input parameters that yield incorrect (and weird) results. Our visual system, while not the same as these systems, is extremely well developed and robust, yet the list of optical illusions that can fool us is quite long. In this study, the optical illusions are really just surprising because they aren't like anything that would fool humans. This isn't to take away from the research; the most interesting result was just how close to valid inputs these erroneously classified images are. But again, this isn't some fatal flaw. This summary completely neglects the fact that the paper also recommends that -- just like distorted images are added to training sets today (you wouldn't want something common like optical aberration from the camera lens screwing up your classifier) -- in the future, these adversarial examples should be added to training sets to mitigate their effects. > In some sense, what we describe is a way to traverse the manifold represented by the network in an efficient way (by optimization) and finding adversarial examples in the input space. The adversarial examples represent low-probability (high-dimensional) “pockets” in the manifold, which are hard to efficiently find by simply randomly sampling the input around a given example. Already, a variety of recent state of the art computer vision models employ input deformations during training for increasing the robustness and convergence speed of the models [9, 13]. These deformations are, however, statistically inefficient, for a given example: they are highly correlated and are drawn from the same distribution throughout the entire training of the model. We propose a scheme to make this process adaptive in a way that exploits the model and its deficiencies in modeling the local space around the training data.[1] [1] http://cs.nyu.edu/~zaremba/docs/understanding.pdf http://cs.nyu.edu/~zaremba/docs/understanding.pdf
- dave_sullivan 12y ago
  > This summary completely neglects the fact that the paper also recommends that -- just like distorted images are added to training sets today (you wouldn't want something common like optical aberration from the camera lens screwing up your classifier) -- in the future, these adversarial examples should be added to training sets to mitigate their effects. I've seen several articles citing this paper as proof that deep learning is deeply flawed, yet they all seem to miss the point you make above. The other interesting result is that the neurons are not in fact individual features you can just grab and drop into another algo--the entire space defined by the model works together, through all the layers. Honestly, that was a more interesting result for me, although I don't know that it negates anything, I've just got to stop telling people that the individual units are features. Not that deep learning is the end-all-be-all of machine learning--it's not. It's just that this paper isn't saying what reporters are saying it's saying... As per usual? Never let the truth get in the way of a good story...
orasis 12y ago
This actually kind of freaks me out. Might it be possible that there is a way to corrupt brains?
- orasis 12y ago
  We are well aware of various optical illusions, but are there "illusions" that be applied against other brain structures, such as memory or beliefs?
  - Someone 12y ago
    There are various haptic illusions (http://en.wikipedia.org/wiki/Haptic_Illusion http://en.wikipedia.org/wiki/Haptic_Illusion) In audio, we have the Shepard tone (http://en.wikipedia.org/wiki/Shepard_tone http://en.wikipedia.org/wiki/Shepard_tone) as an auditory illusion.
  - a-priori 12y ago
    For memory, there's the "Lost in the mall" technique for implanting false memories. It exploits an effect known as memory conformity where people's memories of an event tend to converge after discussing it together. http://en.wikipedia.org/wiki/Lost_in_the_mall_technique http://en.wikipedia.org/wiki/Lost_in_the_mall_technique http://en.wikipedia.org/wiki/Memory_conformity http://en.wikipedia.org/wiki/Memory_conformity Not sure what you mean by beliefs. There was also an experiment on split-brain patients (the connection between the left and right hemispheres is severed) where they'd show a command like "WALK" to the patient's right hemisphere only. They'd get up and walk. But since language is often localized to the left hemisphere, if you talk to them you are talking to the left hemisphere only, which did not see the command. Instead of saying something like "I don't know", they would make up a plausible reason to get up like "I'm getting a drink". Does that count? It exploits what we know about the visual system (half of the visual field goes to each hemisphere) and localization of a particular function (language) in a patient with a specific disability (their corpus collosum is severed, which is mostly asymptomatic) to produce a completely strange result (invention of a motive).
- DanBC 12y ago
  There are experiments that can create a false memory. Not sure if you consider that "corruption". http://psych.hanover.edu/JavaTest/CLE/Cognition/Cognition/falsememory_instructions.html http://psych.hanover.edu/JavaTest/CLE/Cognition/Cognition/fa...
dirkc 12y ago
I don't know if this applies to other classifying algorithms, but I guess this will mean better CAPTCHAs?
peter-row 12y ago
Can you fix it by adding random gaussian noise to the input?
vonnik 12y ago
This is old news. And not really that shocking. You always use multiple models and check them against each other. None is perfect. Big deal.

110 points by vkhuc 12y ago

But then we wouldn't see faces in clouds... Neutral networks are not perfect solutions, they are solutions that get an organism to reproduce successfully. Read any book on color vision humans have similar problems, yet for the most part we see things, and realize that clouds are just clouds and not faces, except for the religious, they lose their shit when faces appear in clouds.

jrochkind1 12y ago

First thought: Can I turn all digital pictures of me into 'adversarial examples', so the eye of sauron can't identify me from pictures? I'm sure it's not as simple as that, presumably any algorithmic modification to an 'adversarial' nature can be countered by other algorithms. But I predict a new realm of 'arms race' here in the future.

cpa 12y ago
Adversarial examples are tied to a specific algorithm, they didn't produce any universal adversarial examples from what I understand.
- tormeh 12y ago
  If I understood it correctly, they looked at the activation of the individual high-level neurons and edited the image such that the high-level activation values changed a lot while the image was still identical to the human eye.
- streptomycin 12y ago
  From the article: What’s more surprising is that the same perturbation can cause a different network, which was trained on a different training dataset, to misclassify the same image. It means that adversarial examples are somewhat universal.
  - cpa 12y ago
    Indeed, but they change the dataset and keep the same algorithm. I didn't see anything about changing the algorithm in itself. That's not very surprising: if you have enough training data, any large training set will be typical because it'll be average enough. But when you choose a specific algorithm, you can exploit its weaknesses and throwing more data won't change anything. Change the algorithm will, though.
    - streptomycin 12y ago
      But why would different training data lead to the same error? I could imagine it would lead to something with the same type of flaw, but why do the same exact adversarial images work, out of the near infinite number of possible images? Doesn't intuitively make sense to me, but I can't say I have much of a background in machine learning. Like if you fit 5 close-to-linear 2d points with a 4th order polynomial, you'll overfit. Change the data slightly and you'll still overfit, but your fit will be very different.

nikhizzle 12y ago

I wonder if subtle make up could be used to make a real life face into an adversarial example.

tormeh 12y ago
You can apply facial makeup that makes your face look completely different to a human observer. Trying to understand how a person with full-coverage facial makeup really looks like is a very frustrating experience. You're trying to see the contours of the face, but your brain keeps focusing on the applied colors. The same makeup would probably work well on a computer as well.
admsyn 12y ago
Yes, actually http://cvdazzle.com/ http://cvdazzle.com/

agibsonccc 12y ago

I think just like any machine learning algorithm, especially with computer vision, you need to prepare things properly. More robust data and images such as moving window on image patches and image rotations, even going as far as applying filters like sobel for edge detection will make your algorithm a lot better. Any algorithms have weaknesses. It's a matter of being aware of them and compensating for them in your model, possibly by using another model.

JabavuAdams 12y ago
The value proposition of deep learning is to eliminate these kind of hand-coded features and to discover the features automagically. However, maybe there's a middle ground. I.e. maybe we don't need the more esoteric features that SIFT uses, but it just makes sense to do edge detection, and say a Fourier transform for audio.
- Punoxysm 12y ago
  The true value proposition of deep learning is not to avoid hand-coded features, but to make better use of scale in data and computational resources. More specifically, adding SIFT or edge detection to your raw pixel input will almost always strictly improve a deep model's performance (though they might be redundant) at a not-particularly-large computational cost. It wouldn't solve the adversarial example problem though, except to the extent that it makes calculating gradients harder.
- joe_the_user 12y ago
  The value proposition of deep learning is to eliminate these kind of hand-coded features and to discover the features automagically. Yes and just to clarify - this seems like an indication that however useful, deep learning can't follow through with that promise. And I think the ability to do this the key thing - all the approaches work at some level but without this "automagicity", each become hostage to brow-wrinkling experts who become the only one to understand the black-magic of algorithm tuning.
- agibsonccc 12y ago
  I wrote [1], I'm plenty aware of the "feature discovery" that goes on, but it can still be an enhancer. See my recent talk[2] for a good overall idea of the situation. Normalization and other data transforms are still required for discovery of features. [1]: http://deeplearning4j.org/ http://deeplearning4j.org/ [2]: https://www.youtube.com/watch?v=hykoKDl1AtE https://www.youtube.com/watch?v=hykoKDl1AtE

kilbasa 12y ago

IIRC even the human brain has the 'adversarial' image flaw (these images will be unique to each person), but one simple workaround is to alter the input image via eye movement (which happens unconsciously).

codeflo 12y ago
IIRC? Can you provide any source or example? This sounds very interesting.
- kilbasa 12y ago
  There was a discussion on slashdot (take it with a grain of salt perhaps) about a similar article. http://slashdot.org/story/14/05/27/1326219/the-flaw-lurking-in-every-deep-neural-net http://slashdot.org/story/14/05/27/1326219/the-flaw-lurking-... The comment I recalled was written by someone with the handle "presidenteloco".

jal278 12y ago

The key claim, from the original paper: > Second, we find that deep neural networks learn input-output mappings that are fairly discontinuous [...] Specifically, we find that we can cause the network to misclassify an image by applying a certain imperceptible perturbation [...] the same perturbation can cause a different network that was trained on a different subset of the dataset, to misclassify the same input. It's an interesting outcome -- but there are many deep-learning approaches and many different benchmarks, so it will be important to see if this is a misleading anecdote or indicative of a systematic problem. [1] http://cs.nyu.edu/~zaremba/docs/understanding.pdf http://cs.nyu.edu/~zaremba/docs/understanding.pdf

varelse 12y ago

This problem was observed 20+ years ago with linear models used for protein structure prediction. For any given model of what described a properly folded protein, one could locate conformations of the same protein that were rated as folded even better than the correct conformation (I called them doppelgangers, but the name "decoy" is what caught on). The statistical naivete of the field led to all sorts of inadvertent mixing of training and test set data which generated a lot of spurious claims for solving the problem. That is until one attempted to find those decoys and they were always found. This led to the creation of the CASP competition to weed this out and the field finally moved forward. http://en.wikipedia.org/wiki/CASP http://en.wikipedia.org/wiki/CASP The key similarity to what I described above is that adversarial search is done posterior to the training of the deep neural network. That makes all the difference in the world IMO. These adversaries may just be strange, otherwise hard to reach bad neighborhoods in image space without using a roadmap. Or they may be an unvaoidable consequence of the curse of dimensionality. http://en.wikipedia.org/wiki/Curse_of_dimensionality http://en.wikipedia.org/wiki/Curse_of_dimensionality But given that neural networks have a gradient, it doesn't shock me that it can serve as a roadmap to locate a set of correlated but seemingly minor changes to an example in order to flip its classification. Doing so is simply back-propagation with constant weight values to propagate the gradient to the input data itself - literally a couple lines of code. IMO there are two interesting experiments to do next (not that anyone will take this seriously I expect, but ya know, hear me now, believe me later): 1. Characterize the statistical nature of the changes in input images and then use those summary statistics as the basis of an image altering algorithm to see if that can be used to flip the classification of any image on its own. If it can, be afraid, your driverless car may have blind spots. If not, then this is probably just a narrower form of overfitting. 2. If it's likely overfitting, attempt an expectation maximization-like fix to the problem. Train the network. Generate adversaries, Add them to the training set, train again and then lather rinse repeat until either the network can't be trained or the problem goes away. Expensive? Yes. But you're Google/Facebook/Microsoft and you have lots of GPUs. No excuses... Failing that, the above is on my todo list so I'm throwing it out there to see if anyone can poke holes in the approach.

magicalist 12y ago
> If it's likely overfitting, attempt an expectation maximization-like fix to the problem. Train the network. Generate adversaries, Add them to the training set, train again and then lather rinse repeat until either the network can't be trained or the problem goes away. As I quoted in my other comment, the paper suggests doing exactly that.

iandanforth 12y ago

Let's not forget that the word "imperceptible" is a heavily laden term in this context. There are numerous modifications to the data that would be "imperceptible" to a machine learning system, but would completely confuse a human. For example if you were to divide the image into a grid, and shuffle the squares, many ML systems would be tolerant to this kind of modification because some training regimes do this anyway. To that system you haven't changed anything important about the image and it would correctly classify it. What this result says to me is that there are really useful features of the data that can successfully classify images that humans are totally unaware of! And that's neat.

segmondy 12y ago
I disagree with your opinion. What this says to me is that DNN is not how humans classifies images.
- maaku 12y ago
  Which would be defeating a strawman. I don't know anyone who claims DNN is precisely how any aspect of the human brain works.
galvan 12y ago
In the linked paper [0], they actually tested that point: after applying Gaussian noise to samples, the model could still recognize them half the time[1], despite being nearly unreadable (to me). [0] http://cs.nyu.edu/~zaremba/docs/understanding.pdf http://cs.nyu.edu/~zaremba/docs/understanding.pdf [1] http://puu.sh/9B4eG/1e9f7eb56b.png http://puu.sh/9B4eG/1e9f7eb56b.png

hyperion2010 12y ago

Not an example from deep learning, but [1] also demonstrates that Bayesian systems also have similar problems with sensitivity to initial conditions that are quite similar. It is also rather striking that these DLNs seem to be tricked by what we would typically think of as noise. 1. http://arxiv.org/abs/1308.6306 http://arxiv.org/abs/1308.6306

hyperbovine 12y ago
It is not so striking when you consider how the typical published "result" in deep learning is obtained -- spend a few months turning all the different knobs that these models have to offer (while possibly inventing a few new ones) until its performance meets or beats the current state of the art on MNIST, CIFAR, and related benchmarks. Which is to say, these models heavily overfit to a few datasets; it should not come as a big surprise when they generalize poorly. What happens now will be that this perturbed dataset will be added to the standard training corpus and the DLNs will become robust to this effect. Then someone will figure out new way to mess them up, and the process will repeat.
- anonymousDan 12y ago
  Would you classify this as progress?
- wantrepeneur 12y ago
  I don't know what papers you are reading, but you seem to have a very distorted view of the literature. How do results on image net, production speech recognition datasets, language modeling, and high resolution satellite images fit the pattern you allege? Once again HN comment quality is depressingly low when it comes to machine learning topics. But I guess to be fair the description in the original link is very confused as well and misunderstands the conclusions of the paper.
apw 12y ago
I thought the Owhadi et al. paper was about model mispecification; i.e., the true model is not in the hypothesis space. That's pretty fundamentally different--and far less of a problem--than gradient descent's "sensitivity to initial conditions".

magicalist 12y ago

The concept is very cool, but it's not surprising that dimensionality reduction through a non-linear process is going to result in sections of input parameters that yield incorrect (and weird) results. Our visual system, while not the same as these systems, is extremely well developed and robust, yet the list of optical illusions that can fool us is quite long. In this study, the optical illusions are really just surprising because they aren't like anything that would fool humans. This isn't to take away from the research; the most interesting result was just how close to valid inputs these erroneously classified images are. But again, this isn't some fatal flaw. This summary completely neglects the fact that the paper also recommends that -- just like distorted images are added to training sets today (you wouldn't want something common like optical aberration from the camera lens screwing up your classifier) -- in the future, these adversarial examples should be added to training sets to mitigate their effects. > In some sense, what we describe is a way to traverse the manifold represented by the network in an efficient way (by optimization) and finding adversarial examples in the input space. The adversarial examples represent low-probability (high-dimensional) “pockets” in the manifold, which are hard to efficiently find by simply randomly sampling the input around a given example. Already, a variety of recent state of the art computer vision models employ input deformations during training for increasing the robustness and convergence speed of the models [9, 13]. These deformations are, however, statistically inefficient, for a given example: they are highly correlated and are drawn from the same distribution throughout the entire training of the model. We propose a scheme to make this process adaptive in a way that exploits the model and its deficiencies in modeling the local space around the training data.[1] [1] http://cs.nyu.edu/~zaremba/docs/understanding.pdf http://cs.nyu.edu/~zaremba/docs/understanding.pdf

dave_sullivan 12y ago
> This summary completely neglects the fact that the paper also recommends that -- just like distorted images are added to training sets today (you wouldn't want something common like optical aberration from the camera lens screwing up your classifier) -- in the future, these adversarial examples should be added to training sets to mitigate their effects. I've seen several articles citing this paper as proof that deep learning is deeply flawed, yet they all seem to miss the point you make above. The other interesting result is that the neurons are not in fact individual features you can just grab and drop into another algo--the entire space defined by the model works together, through all the layers. Honestly, that was a more interesting result for me, although I don't know that it negates anything, I've just got to stop telling people that the individual units are features. Not that deep learning is the end-all-be-all of machine learning--it's not. It's just that this paper isn't saying what reporters are saying it's saying... As per usual? Never let the truth get in the way of a good story...

orasis 12y ago

This actually kind of freaks me out. Might it be possible that there is a way to corrupt brains?

orasis 12y ago
We are well aware of various optical illusions, but are there "illusions" that be applied against other brain structures, such as memory or beliefs?
- Someone 12y ago
  There are various haptic illusions (http://en.wikipedia.org/wiki/Haptic_Illusion http://en.wikipedia.org/wiki/Haptic_Illusion) In audio, we have the Shepard tone (http://en.wikipedia.org/wiki/Shepard_tone http://en.wikipedia.org/wiki/Shepard_tone) as an auditory illusion.
- a-priori 12y ago
  For memory, there's the "Lost in the mall" technique for implanting false memories. It exploits an effect known as memory conformity where people's memories of an event tend to converge after discussing it together. http://en.wikipedia.org/wiki/Lost_in_the_mall_technique http://en.wikipedia.org/wiki/Lost_in_the_mall_technique http://en.wikipedia.org/wiki/Memory_conformity http://en.wikipedia.org/wiki/Memory_conformity Not sure what you mean by beliefs. There was also an experiment on split-brain patients (the connection between the left and right hemispheres is severed) where they'd show a command like "WALK" to the patient's right hemisphere only. They'd get up and walk. But since language is often localized to the left hemisphere, if you talk to them you are talking to the left hemisphere only, which did not see the command. Instead of saying something like "I don't know", they would make up a plausible reason to get up like "I'm getting a drink". Does that count? It exploits what we know about the visual system (half of the visual field goes to each hemisphere) and localization of a particular function (language) in a patient with a specific disability (their corpus collosum is severed, which is mostly asymptomatic) to produce a completely strange result (invention of a motive).
DanBC 12y ago
There are experiments that can create a false memory. Not sure if you consider that "corruption". http://psych.hanover.edu/JavaTest/CLE/Cognition/Cognition/falsememory_instructions.html http://psych.hanover.edu/JavaTest/CLE/Cognition/Cognition/fa...

dirkc 12y ago

I don't know if this applies to other classifying algorithms, but I guess this will mean better CAPTCHAs?

peter-row 12y ago

Can you fix it by adding random gaussian noise to the input?

vonnik 12y ago

This is old news. And not really that shocking. You always use multiple models and check them against each other. None is perfect. Big deal.