5 ms·

What do you think of alternative, non-neural-network approaches to deep learning such as Bayesian networks[1] or Support Vector Machines (which as I understand

by PeCaN 11y ago

What do you think of alternative, non-neural-network approaches to deep learning such as Bayesian networks[1] or Support Vector Machines (which as I understand it are largely dead, but the theory behind them is interesting and could lead to more developments). Is the future of deep learning just massive neural networks on huge GPU clusters? Is anyone big doing research on new approaches, or just focusing on fixing the problems with neural networks? (Which, given that the whole idea of deep NNs is rather new anyway, I suppose it might be a little early to look at other algorithms.) 1. http://www.psy.cmu.edu/~ckemp/papers/maask09.pdf http://www.psy.cmu.edu/~ckemp/papers/maask09.pdf

jimbokun 11y ago
"Support Vector Machines (which as I understand it are largely dead, but the theory behind them is interesting and could lead to more developments)." Wow, that happened fast. SVMs seemed all the rage a few years ago. I think it shows the importance of understanding the formalisms underlying machine learning approaches, and their corresponding strengths and weaknesses. Neural networks had been on the outs for a long time, before the realization that having much more data to train against could massively increase their usefulness. Wonder what the next ML technique to see a resurgence will be.
jph00 11y ago
The difference with deep neural networks is that they can do so much of their own feature generation. It's very different to bayesian networks or SVMs. Because they can create their own features (to a large extent) they can handle unstructured data more simply and more accurately. Neural nets are dramatically improving every month. Recently, the major developments have been around using a wider variety of differentiable functions as layers in the network - see for example Spatial Transformer Networks, and also the more general architectures such as Neural Turing Machines.

by PeCaN 11y ago

jimbokun 11y ago

"Support Vector Machines (which as I understand it are largely dead, but the theory behind them is interesting and could lead to more developments)." Wow, that happened fast. SVMs seemed all the rage a few years ago. I think it shows the importance of understanding the formalisms underlying machine learning approaches, and their corresponding strengths and weaknesses. Neural networks had been on the outs for a long time, before the realization that having much more data to train against could massively increase their usefulness. Wonder what the next ML technique to see a resurgence will be.

jph00 11y ago

The difference with deep neural networks is that they can do so much of their own feature generation. It's very different to bayesian networks or SVMs. Because they can create their own features (to a large extent) they can handle unstructured data more simply and more accurately. Neural nets are dramatically improving every month. Recently, the major developments have been around using a wider variety of differentiable functions as layers in the network - see for example Spatial Transformer Networks, and also the more general architectures such as Neural Turing Machines.