7 ms·
Understanding LSTM networks
- mistermaster 11y agogreat explanation. many thanks! hochreiter is genius!
- colah3 11y ago> hochreiter is genius! It's really impressive to me how farsighted his work on LSTMs was. In any case, I'm glad you liked the post.
- p1esk 11y agoMichael, this post nicely completes your book about neural networks. I was a little surprised you didn't write it yourself.
- ambicapter 11y agoAnybody know what he uses for his diagrams?
- colah3 11y agoAll of the diagrams in this post were made in Inkscape, with the LaTeX plugin for equations.
- reader5000 11y agoWhats amazing to me is that if I understand correctly backprop still works. It is very odd that SGD on the error function for some training data is conceptually equivalent to teaching all the gates for each hidden feature when to open/close given the next input in a sequence.
- colah3 11y ago> Whats amazing to me is that if I understand correctly backprop still works. Yep! One computes the gradient with backpropagation and trains LSTMs on that. > It is very odd that SGD on the error function for some training data is conceptually equivalent to teaching all the gates for each hidden feature when to open/close given the next input in a sequence. Agreed, it's pretty remarkable the things one can learn with gradient descent. I'd like to understand this better.
- hyperbovine 11y agoI don't understand why this is remarkable, could you elaborate? You have a smooth map from inputs to outputs, you differentiate it via automated applications of the chain rule and use this to make the error function smaller. Why should this be remarkable?
- reader5000 11y agoAn LSTM is slightly different than a standard feedforward unit. LSTM unit has internal state dependent on past inputs. Gradient descent trains the LSTM not only which output for a given input, but also how to update the internal state given an input. That the whole thing should still be differentiable is not intuitive to me. edit: It's kind of like, if not actually equivalent to, programming a Turing machine by its gradient on training data.
- colah3 11y agoIf I didn't know that neural nets worked, and someone explained the idea, I'd strongly expect them to get stuck in local minima, instead of learning this elaborate behavior. (That's basically what happens with a normal RNN, I think.) It's probably a case of my intuitions about optimization being really broken in high-dimensional spaces, but I'd like to improve that. I'd also like to understand how the cost surface we're optimizing on interacts with network architecture decisions. Both, unfortunately, are very hard demands.
- hyperbovine 11y agoUnless there is some major advance in nonconvex optimization here, (sub)gradient descent on the function you describe in your post is almost certainly converging to a local minimum as well. I guess maybe the surprise is that local minima can perform well? But again, this does not seem like such a surprise when you consider the number of parameters being fitted. I do not have much experience with LSTMs per se, but modern i.e. conv-nets are basically O(10^8)-dimensional nonlinear maps. It just doesn't seem all that unexpected that you can embed a huge number of patterns into such an object.
- 11y ago
- ot 11y agocolah, your posts combine a deep level of understanding and an exceptional clarity. These are both rare, especially in the cargo-cult-driven world of neural networks and deep learning. I hope you keep writing as much as you can. Thanks!
- d136o 11y agoThanks colah, that was a very readable walk-through. I've been making my way through Bishop's PRML ch 5 to get as much of a handle as possible on NNs, but your intro here to LSTM's makes me want to jump ahead and skip to the new stuff :)