4 ms·
Actually, as a great fan of what has come out of the current deep learning hypefest, I'd question whether "deep" really matters. Most of the great successes hav
by ewjordan 10y ago
Actually, as a great fan of what has come out of the current deep learning hypefest, I'd question whether "deep" really matters. Most of the great successes have resulted from medium depth nets using the same shitty backprop algorithms that have been known for decades.
"Deep learning" is a red herring. They're just doing exactly what the shallow learning pioneers told them to do twenty years ago, just with more computing power.
- riyadparvez 10y agoThere are a lot of innovations recently, not just the computing power -- layer by layer unsupervised training, batch normalization, ReLU, dropouts, momentum, gated memory unit, neural language modeling, encoder-decoder just to name a few.
- ewjordan 10y agoYep, there are a lot of innovations, and plenty that people are holding back as well. None of it involves the word "deep", it's all just bog standard 1980 style nets with tweaks.
- rcpt 10y agoGood iclr paper on this "Do Deep Nets Really Need to be Deep?" http://arxiv.org/abs/1312.6184 http://arxiv.org/abs/1312.6184