4 ms·

[Deleted]

by make3 9y ago

[Deleted]

Scea91 9y ago
> it's gone out of fashion because it's been beaten extremely convincingly by methods that learned everything from scratch, in every single task Totally false. By every single task you probably mean "every deep learning success story" which would be tautological. There are tons of tasks where deep learning doesn't quite work yet and you still have to hand-craft your features. Example from my domain: Try learning a classifier for malware which uses just raw binaries on input.
- nl 9y ago
  My understanding is that work on deep learning for malware classification is pretty early. Having said that, my understanding is that they are getting closer. https://pdfs.semanticscholar.org/f1c8/7533e628fd374b9e98f1e1ccc18a0e8f195d.pdf https://pdfs.semanticscholar.org/f1c8/7533e628fd374b9e98f1e1... https://devblogs.nvidia.com/malware-detection-neural-networks/ https://devblogs.nvidia.com/malware-detection-neural-network...
  - Scea91 9y ago
    Results in this field usually look way better than they would be in a production environment. Regarding articles you mentioned: Using ROC curve for evaluation in this case is a red-flag because it doesn't take the data imbalance into account. Precision-Recall curve is way more suitable. You can have great AUC on the ROC curve but precision can be near zero in highly imbalanced problems such as malware detection. Precision is the probability that a positive detection is a true positive. Which is usually the measure you are most interested in. The problem is that precision changes if you change the class priors. Because of that, the results are always very dataset specific. With that said, I do not say that machine learning or neural networks do not work on this task. It just doesn't work in the end-to-end manner where you just feed raw binaries as input to some generic architecture as we can do with images in some tasks.
YeGoblynQueenne 9y ago
In the video, both LeCun and Manning argue convincingly that this is not the case. LeCun believes that in some idealised future that is stil far away, priors will not be necessary. Both seem to think that, right now, you can't do anything completely from scratch. I wonder what examples you have in mind btw, other than self-play.

[Deleted]

by make3 9y ago

[Deleted]

> it's gone out of fashion because it's been beaten extremely convincingly by methods that learned everything from scratch, in every single task Totally false. By every single task you probably mean "every deep learning success story" which would be tautological. There are tons of tasks where deep learning doesn't quite work yet and you still have to hand-craft your features. Example from my domain: Try learning a classifier for malware which uses just raw binaries on input.

nl 9y ago
My understanding is that work on deep learning for malware classification is pretty early. Having said that, my understanding is that they are getting closer. https://pdfs.semanticscholar.org/f1c8/7533e628fd374b9e98f1e1ccc18a0e8f195d.pdf https://pdfs.semanticscholar.org/f1c8/7533e628fd374b9e98f1e1... https://devblogs.nvidia.com/malware-detection-neural-networks/ https://devblogs.nvidia.com/malware-detection-neural-network...
- Scea91 9y ago
  Results in this field usually look way better than they would be in a production environment. Regarding articles you mentioned: Using ROC curve for evaluation in this case is a red-flag because it doesn't take the data imbalance into account. Precision-Recall curve is way more suitable. You can have great AUC on the ROC curve but precision can be near zero in highly imbalanced problems such as malware detection. Precision is the probability that a positive detection is a true positive. Which is usually the measure you are most interested in. The problem is that precision changes if you change the class priors. Because of that, the results are always very dataset specific. With that said, I do not say that machine learning or neural networks do not work on this task. It just doesn't work in the end-to-end manner where you just feed raw binaries as input to some generic architecture as we can do with images in some tasks.

YeGoblynQueenne 9y ago

In the video, both LeCun and Manning argue convincingly that this is not the case. LeCun believes that in some idealised future that is stil far away, priors will not be necessary. Both seem to think that, right now, you can't do anything completely from scratch. I wonder what examples you have in mind btw, other than self-play.