4 ms·
How about a single layer neural net trained with dropout? Not deep learning because there's only 1 layer, but the technique is fairly new and used in deep learn
by dave_sullivan 11y ago
How about a single layer neural net trained with dropout? Not deep learning because there's only 1 layer, but the technique is fairly new and used in deep learning, popularized by some of the guys that popularized deep learning, usually mentioned in conversations about deep learning. It's really a shallow neural model though. Word2vec is similarly related, but you're right in that it is not a perceptron-based neural network with multiple layers (ie "deep neural network")
Still, interesting blog post, worth reading and googling for more information. FWIW I just use the deep learning definition of "neural net or representation learning research since 2006" and find it fits better.
- deleted 11y ago[deleted]
- fchollet 11y ago> the technique is fairly new and used in deep learning, popularized by some of the guys that popularized deep learning, usually mentioned in conversations about deep learning Logistic regression with regularization is fairly new? 'Pioneered' by the same people as deep convolutional neural networks? Are you certain about this?
- agibsonccc 11y agoI think you need to define regularization. L1/L2? Yes those are old. Drop out itself IS fairly new[1]. As well as its newer cousin drop connect. I agree neural nets themselves are basically just a crazier parametric model. Many of the things we do to modify the gradient are applicable to logistic and other simpler regression techniques as well. [2]: https://cs.nyu.edu/~wanli/dropc/dropc.pdfhttp://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf https://cs.nyu.edu/~wanli/dropc/dropc.pdfhttp://www.cs.toron...