5 ms·

I think just like any machine learning algorithm, especially with computer vision, you need to prepare things properly. More robust data and images such as movi

I think just like any machine learning algorithm, especially with computer vision, you need to prepare things properly. More robust data and images such as moving window on image patches and image rotations, even going as far as applying filters like sobel for edge detection will make your algorithm a lot better. Any algorithms have weaknesses. It's a matter of being aware of them and compensating for them in your model, possibly by using another model.

JabavuAdams 12y ago
The value proposition of deep learning is to eliminate these kind of hand-coded features and to discover the features automagically. However, maybe there's a middle ground. I.e. maybe we don't need the more esoteric features that SIFT uses, but it just makes sense to do edge detection, and say a Fourier transform for audio.
- Punoxysm 12y ago
  The true value proposition of deep learning is not to avoid hand-coded features, but to make better use of scale in data and computational resources. More specifically, adding SIFT or edge detection to your raw pixel input will almost always strictly improve a deep model's performance (though they might be redundant) at a not-particularly-large computational cost. It wouldn't solve the adversarial example problem though, except to the extent that it makes calculating gradients harder.
- joe_the_user 12y ago
  The value proposition of deep learning is to eliminate these kind of hand-coded features and to discover the features automagically. Yes and just to clarify - this seems like an indication that however useful, deep learning can't follow through with that promise. And I think the ability to do this the key thing - all the approaches work at some level but without this "automagicity", each become hostage to brow-wrinkling experts who become the only one to understand the black-magic of algorithm tuning.
- agibsonccc 12y ago
  I wrote [1], I'm plenty aware of the "feature discovery" that goes on, but it can still be an enhancer. See my recent talk[2] for a good overall idea of the situation. Normalization and other data transforms are still required for discovery of features. [1]: http://deeplearning4j.org/ http://deeplearning4j.org/ [2]: https://www.youtube.com/watch?v=hykoKDl1AtE https://www.youtube.com/watch?v=hykoKDl1AtE
  - wantrepeneur 12y ago
    That library looks like garbage. The website describes convnets as a "type of restricted Boltzmann machine." How can anyone trust a library with that level of misinformation?
    - agibsonccc 12y ago
      I appreciate the feedback. I need to clarify the implementation in DL4j in the documentation. The impl is a convolutional RBM. That being said, where's your deep learning library? ;) The documentation is still being worked on. If that's the only thing you can cherry pick from a new project, I must not be doing too bad. Edit: Clarified. I know you're just a troll account, but I'll throw this out there anyways, I think anyone who judges the quality of a lib based on a 2 second reading isn't qualified to judge much. A lot of it does rely on me with the documentation, but I'd love to chat with you one on one if you think I'm not qualified. Trash me all day, I'll either learn something or embarrass you. Both are fine with me ;).
      - wantrepeneur 12y ago
        A lot of people like Torch. I haven't looked at pylearn2 in a while, but that might be good too. Then there are a few researcher libraries with limited documentation. Nitish Srivastava has his DeepNet library and George Dahl has gdbn and there are certainly a few others too. Who knows, maybe people will start contributing documentation. I would probably recommend Torch at this point. The incentives don't exist for the experts to make really good open source projects and spend all the time required maintaining them and helping people with them.
jre 12y ago
The problem raised in the paper is that most of the time, you can safely assume that if you have a new data point that is very close to a bunch of correctly-classified data points and far away from points of different classes, the new data point will be correctly classified. In other words, you assume the classification probability is locally smooth. The problem is that the adversarial examples they are able to come up with are very close to the original images, so this means the smoothness assumption seem to be invalid for deep learning models. As they put it in the paper : "Our main result is that for deep neural networks, the smoothness assumption that underlies many kernel methods does not hold." It's going to be interesting to see what happen when other researchers try to replicate the results for other models and datasets.
- darkmighty 12y ago
  What if they introduced smoothness artifically? You can easily do that by averaging with a certain window function the decision value. To accomplish that, for example, they could take perturbations of the input and then e.g. take a majority majority vote.

JabavuAdams 12y ago

The value proposition of deep learning is to eliminate these kind of hand-coded features and to discover the features automagically. However, maybe there's a middle ground. I.e. maybe we don't need the more esoteric features that SIFT uses, but it just makes sense to do edge detection, and say a Fourier transform for audio.

Punoxysm 12y ago
The true value proposition of deep learning is not to avoid hand-coded features, but to make better use of scale in data and computational resources. More specifically, adding SIFT or edge detection to your raw pixel input will almost always strictly improve a deep model's performance (though they might be redundant) at a not-particularly-large computational cost. It wouldn't solve the adversarial example problem though, except to the extent that it makes calculating gradients harder.
joe_the_user 12y ago
The value proposition of deep learning is to eliminate these kind of hand-coded features and to discover the features automagically. Yes and just to clarify - this seems like an indication that however useful, deep learning can't follow through with that promise. And I think the ability to do this the key thing - all the approaches work at some level but without this "automagicity", each become hostage to brow-wrinkling experts who become the only one to understand the black-magic of algorithm tuning.
agibsonccc 12y ago
I wrote [1], I'm plenty aware of the "feature discovery" that goes on, but it can still be an enhancer. See my recent talk[2] for a good overall idea of the situation. Normalization and other data transforms are still required for discovery of features. [1]: http://deeplearning4j.org/ http://deeplearning4j.org/ [2]: https://www.youtube.com/watch?v=hykoKDl1AtE https://www.youtube.com/watch?v=hykoKDl1AtE
- wantrepeneur 12y ago
  That library looks like garbage. The website describes convnets as a "type of restricted Boltzmann machine." How can anyone trust a library with that level of misinformation?
  - agibsonccc 12y ago
    I appreciate the feedback. I need to clarify the implementation in DL4j in the documentation. The impl is a convolutional RBM. That being said, where's your deep learning library? ;) The documentation is still being worked on. If that's the only thing you can cherry pick from a new project, I must not be doing too bad. Edit: Clarified. I know you're just a troll account, but I'll throw this out there anyways, I think anyone who judges the quality of a lib based on a 2 second reading isn't qualified to judge much. A lot of it does rely on me with the documentation, but I'd love to chat with you one on one if you think I'm not qualified. Trash me all day, I'll either learn something or embarrass you. Both are fine with me ;).
    - wantrepeneur 12y ago
      A lot of people like Torch. I haven't looked at pylearn2 in a while, but that might be good too. Then there are a few researcher libraries with limited documentation. Nitish Srivastava has his DeepNet library and George Dahl has gdbn and there are certainly a few others too. Who knows, maybe people will start contributing documentation. I would probably recommend Torch at this point. The incentives don't exist for the experts to make really good open source projects and spend all the time required maintaining them and helping people with them.

jre 12y ago

The problem raised in the paper is that most of the time, you can safely assume that if you have a new data point that is very close to a bunch of correctly-classified data points and far away from points of different classes, the new data point will be correctly classified. In other words, you assume the classification probability is locally smooth. The problem is that the adversarial examples they are able to come up with are very close to the original images, so this means the smoothness assumption seem to be invalid for deep learning models. As they put it in the paper : "Our main result is that for deep neural networks, the smoothness assumption that underlies many kernel methods does not hold." It's going to be interesting to see what happen when other researchers try to replicate the results for other models and datasets.

darkmighty 12y ago
What if they introduced smoothness artifically? You can easily do that by averaging with a certain window function the decision value. To accomplish that, for example, they could take perturbations of the input and then e.g. take a majority majority vote.