5 ms·
Not sure why this is so highly upvoted. Nobody is questioning that deep networks work better than shallow ones, and there is a good understanding in academia of
by argonaut 10y ago
Not sure why this is so highly upvoted. Nobody is questioning that deep networks work better than shallow ones, and there is a good understanding in academia of why (that fits with most lay people's intuition). I hardly consider that the most interesting or relevant question.
- ewjordan 10y agoActually, as a great fan of what has come out of the current deep learning hypefest, I'd question whether "deep" really matters. Most of the great successes have resulted from medium depth nets using the same shitty backprop algorithms that have been known for decades. "Deep learning" is a red herring. They're just doing exactly what the shallow learning pioneers told them to do twenty years ago, just with more computing power.
- riyadparvez 10y agoThere are a lot of innovations recently, not just the computing power -- layer by layer unsupervised training, batch normalization, ReLU, dropouts, momentum, gated memory unit, neural language modeling, encoder-decoder just to name a few.
- ewjordan 10y agoYep, there are a lot of innovations, and plenty that people are holding back as well. None of it involves the word "deep", it's all just bog standard 1980 style nets with tweaks.
- rcpt 10y agoGood iclr paper on this "Do Deep Nets Really Need to be Deep?" http://arxiv.org/abs/1312.6184 http://arxiv.org/abs/1312.6184