25 ms·

Ask HN: What's the big deal about Deep Learning?

8 points by thewarrior 13y ago

PaulHoule 13y ago
It's a big deal because it works. Back in the 1960's people only knew how to train one-layer neural networks. By the late 1970's and 1980's people were training "multi-layer perceptrons" but it wasn't practical to train networks with a lot of layers. In the 1990's and early 2000's Neural Networks were out of favor for academic work, not so much that they were bad but for almost any learning problem there was some other kind of algorithm that did better. Neural Networks were still used a lot in commercial applications that weren't so talked about since they thrive given the kind of intensive attention that they sometimes get in such environments. Geoff Hinton then figured out a way to quickly train deep networks to learn the statistical character of the input data. This works as an unsupervised process. It's kindof like the first layer learns very simple features that recur in the data and then the second layer learns relationships between theem and so forth. After you've done that, you can teach the network to do some specific task using a supervised training algorithm such as back propagation. Because the network is close to where it needs to be, you can now apply back propagation to the whole network with excellent results. Deep learning is such an advance that neural networks are now often the leading algorithm for many learning problems.
agibsonccc 13y ago
Simple: Data Scientists spend a great deal of time doing feature engineering. That is, finding observations about their data that are indicative of a prediction they are trying to make or a piece of data (say: a user or an image, perhaps a document) they are trying to label. Deep Learning handles this for us. Typically machine learning takes a lot of domain knowledge. Deep Learning is capable of automatically finding correlations in the data and allowing us to focus on solving problems rather than worrying about what good indicators of data are. Re: quick pretraining. Try again: deep learning pretraining takes forever to train due to how many passes through the dataset you need to make in order for it to make good approximations of the data. That being said: deep learning is very receptive to map reduce type algorithms and online learning (streaming of data in mini batches) That's why bigger companies tend to use it at scale. That being said: it's astonishingly accurate despite being hard to train. I will be more than glad to answer specific questions if you'd like. I'll give you a few use cases: Google uses it in their voice recognition on android. IBM uses it in their Watson technology. Google uses it in their Google+ image search. It's being used for sentiment analysis at stanford. Source: I'm the author of a newer deep learning framework called deeplearning4j [1]. My goal with this is to make it easier to train these neural networks by just say: feeding it images and all you need to do is specify the labels: or text. I'm also a collaborator/committer on another effort called metronome which is deeplearning on hadoop [2]. If you'd like an example of what deep learning is capable of, please take a look at [3]. It's a very different way of doing machine learning. [1] http://deeplearning4j.org http://deeplearning4j.org [2] https://github.com/jpatanooga/Metronome https://github.com/jpatanooga/Metronome [3] https://jpatanooga.github.io/Metronome/rbm20140306.html https://jpatanooga.github.io/Metronome/rbm20140306.html
ACow_Adonis 13y ago
Are there any resources for someone wanting to get started looking into coding/understanding deep learning algorithms? By which i mean, i'm already a data scientist. I'm not looking for software that runs it for me per se, i'm looking for resources that could let me implement/research it myself. I've seen a lot of hype. Some of the claims I've seen people make for deep learning I know must be objectively false and sound like a lot of the hype/marketing for other machine learning techniques, but the ideas also look somewhat promising, so i'm really curious to get involved and at least be able to make an educated decision...
- sytelus 13y ago
  The http://deeplearning.net/ http://deeplearning.net/ is a go to website for beginners. You would find everything from datasets to experiment to tutorials. This subject requires a lot of match and ML background though and would need sometime to catch up to the latest research. Coding part will involve GPU code if you want to go truely in with large data sets. If you want to just apply the algorithms to specific problem without diving deep in to its theory there are few kits available in Python and C++. There a tech talks and other videos on tutorial page that might be good starting point.
sytelus 13y ago
Current "machine learning" algorithms such as decision trees, SVM, regression etc are in a way cheating. In all these algorithms you need to design features and they contain much of the smarts that you see from "machine learning" algorithms. In other words, an impressive machine learned model is typically a result of an impressive features gymnastic manually programmed by humans after lot of sweat and blood. A true machine learning algorithms shouldn't require humans to encode their smarts and knowledge. If these algorithms existed then they would just look at stream of symbols or sensory inputs that occurs in the nature and extract patterns from it. Deep learning gets closer to this by eliminating need for developing features (to some extent so far). I think it is first of many steps towards true machine learning.

8 points by thewarrior 13y ago

It's a big deal because it works. Back in the 1960's people only knew how to train one-layer neural networks. By the late 1970's and 1980's people were training "multi-layer perceptrons" but it wasn't practical to train networks with a lot of layers. In the 1990's and early 2000's Neural Networks were out of favor for academic work, not so much that they were bad but for almost any learning problem there was some other kind of algorithm that did better. Neural Networks were still used a lot in commercial applications that weren't so talked about since they thrive given the kind of intensive attention that they sometimes get in such environments. Geoff Hinton then figured out a way to quickly train deep networks to learn the statistical character of the input data. This works as an unsupervised process. It's kindof like the first layer learns very simple features that recur in the data and then the second layer learns relationships between theem and so forth. After you've done that, you can teach the network to do some specific task using a supervised training algorithm such as back propagation. Because the network is close to where it needs to be, you can now apply back propagation to the whole network with excellent results. Deep learning is such an advance that neural networks are now often the leading algorithm for many learning problems.

agibsonccc 13y ago

Simple: Data Scientists spend a great deal of time doing feature engineering. That is, finding observations about their data that are indicative of a prediction they are trying to make or a piece of data (say: a user or an image, perhaps a document) they are trying to label. Deep Learning handles this for us. Typically machine learning takes a lot of domain knowledge. Deep Learning is capable of automatically finding correlations in the data and allowing us to focus on solving problems rather than worrying about what good indicators of data are. Re: quick pretraining. Try again: deep learning pretraining takes forever to train due to how many passes through the dataset you need to make in order for it to make good approximations of the data. That being said: deep learning is very receptive to map reduce type algorithms and online learning (streaming of data in mini batches) That's why bigger companies tend to use it at scale. That being said: it's astonishingly accurate despite being hard to train. I will be more than glad to answer specific questions if you'd like. I'll give you a few use cases: Google uses it in their voice recognition on android. IBM uses it in their Watson technology. Google uses it in their Google+ image search. It's being used for sentiment analysis at stanford. Source: I'm the author of a newer deep learning framework called deeplearning4j [1]. My goal with this is to make it easier to train these neural networks by just say: feeding it images and all you need to do is specify the labels: or text. I'm also a collaborator/committer on another effort called metronome which is deeplearning on hadoop [2]. If you'd like an example of what deep learning is capable of, please take a look at [3]. It's a very different way of doing machine learning. [1] http://deeplearning4j.org http://deeplearning4j.org [2] https://github.com/jpatanooga/Metronome https://github.com/jpatanooga/Metronome [3] https://jpatanooga.github.io/Metronome/rbm20140306.html https://jpatanooga.github.io/Metronome/rbm20140306.html

ACow_Adonis 13y ago

Are there any resources for someone wanting to get started looking into coding/understanding deep learning algorithms? By which i mean, i'm already a data scientist. I'm not looking for software that runs it for me per se, i'm looking for resources that could let me implement/research it myself. I've seen a lot of hype. Some of the claims I've seen people make for deep learning I know must be objectively false and sound like a lot of the hype/marketing for other machine learning techniques, but the ideas also look somewhat promising, so i'm really curious to get involved and at least be able to make an educated decision...

sytelus 13y ago
The http://deeplearning.net/ http://deeplearning.net/ is a go to website for beginners. You would find everything from datasets to experiment to tutorials. This subject requires a lot of match and ML background though and would need sometime to catch up to the latest research. Coding part will involve GPU code if you want to go truely in with large data sets. If you want to just apply the algorithms to specific problem without diving deep in to its theory there are few kits available in Python and C++. There a tech talks and other videos on tutorial page that might be good starting point.

sytelus 13y ago

Current "machine learning" algorithms such as decision trees, SVM, regression etc are in a way cheating. In all these algorithms you need to design features and they contain much of the smarts that you see from "machine learning" algorithms. In other words, an impressive machine learned model is typically a result of an impressive features gymnastic manually programmed by humans after lot of sweat and blood. A true machine learning algorithms shouldn't require humans to encode their smarts and knowledge. If these algorithms existed then they would just look at stream of symbols or sensory inputs that occurs in the nature and extract patterns from it. Deep learning gets closer to this by eliminating need for developing features (to some extent so far). I think it is first of many steps towards true machine learning.