5 ms·
great writeup! in general, to make deep learning faster avoid using backpropagation
by bra-ket 12y ago
great writeup!
in general, to make deep learning faster avoid using backpropagation
- iskander 12y agoWhat else do you use to train the network?
- bra-ket 12y agorandom matrices + svd: http://www.ntu.edu.sg/home/egbhuang/ http://www.ntu.edu.sg/home/egbhuang/
- gcr 12y agoI would be very interested to see if this approach outperforms more conventional deep networks trained with backpropagation. It's important to note that (from what I understand) this paper seems to focus on building single-layer NNs, which are quite different from the behemoth AlexNet that this article focuses on.
- deleted 12y ago[deleted]
- Fede_V 12y agoVery early on, people experimented heavily with non-gradient based optimization (genetic algorithms and the like) to optimize neural networks, but the results were rather poor. For very large parameter spaces, gradient based optimization is really the only game in town. There's also lots of clever tricks (gradient clipping, momentum, adagrad, etc) that are involved in optimizing the convergence.
- gcr 12y agoThere's a bit of confusion here, because backpropagation is only used to train the network. After your network is trained, you don't want to change it, so you wouldn't run backpropagation in production. As for making training faster, second-order methods don't generally work as well as backpropagation for large networks. Some notes from Yann LeCunn! http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf