4 ms·

Understanding LSTM Networks (2015)

106 points by mkagenius 9y ago

rahimnathwani 9y ago
If you like this and Chris' other posts, don't forget to check out https://distill.pub/ https://distill.pub/ which he co-founded. Such high quality explanations.
amelius 9y ago
Too hand-wavy for my taste. I have the feeling I still don't know why they work.
- senatorobama 9y ago
  Agreed. Don't know why this article gets high praise, I'm sure the pretty pictures help.
  - dbranes 9y ago
    I like this because it describes fully and precisely the structure of an LSTM cell, in a way which mostly avoids ML/stats jargon. From this article I can correctly and easily read off the model architecture it describes, as a composition of smooth maps on modules over the real numbers, which is more than what one can say about a lot of papers.
  - colah3 9y ago
    Hi! I'm the author. I was also pretty surprised by the amount of attention this article has gotten. That said, I can give you my take on how it advanced the dialogue. As you note, the main contribution of this article is the diagrams -- but I think it's a bit deeper than being pretty. Previously, diagrams of LSTMs had lots of loops in them. This made them hard to understand. (In fact, they were ambiguous because of unclear execution order.) The diagrams in this article unrolled the loops, which seems to be a much easier way to reason about LSTMs. They also work at the abstraction of layers instead of weights and matrix multiplies, which make the diagram less noisy and focuses the reader on the important ideas. Since diagrams were previously challenging to consume and understand, explanations prior to my article tended to focus on the six equations than define an LSTM. While that's a relatively clear way to think about LSTMs once you've really internalized them, it can be very hard when you are trying to understand them for the first time. In particular, the equations introduce ~15 variables. If you aren't already familiar with those variables, it's extremely challenging to keep them all in your working memory and understand what's going on. In contrast, the diagrams don't require a new reader to hold lots of things in working memory, and visually hint a great deal of the narrative (like the horizontal line at the top).
- hinkley 9y ago
  It takes him 12 paragraphs to define the acronym in the title.
rahulchhabra07 9y ago
Had been my first reading on LSTM and was so smooth and lucid. I did need to read more literature and code, but a great resource to begin with.
blueyes 9y ago
This is also a good one: https://deeplearning4j.org/lstm https://deeplearning4j.org/lstm
stared 9y ago
It's a great text, but I recommend supplementing it with: http://blog.echen.me/2017/05/30/exploring-lstms/ http://blog.echen.me/2017/05/30/exploring-lstms/. Additionally, GRU may be simpler to understand memory unit.
nafizh 9y ago
prev discussion: https://news.ycombinator.com/item?id=10130341 https://news.ycombinator.com/item?id=10130341

106 points by mkagenius 9y ago

If you like this and Chris' other posts, don't forget to check out https://distill.pub/ https://distill.pub/ which he co-founded. Such high quality explanations.

amelius 9y ago

Too hand-wavy for my taste. I have the feeling I still don't know why they work.

senatorobama 9y ago
Agreed. Don't know why this article gets high praise, I'm sure the pretty pictures help.
- dbranes 9y ago
  I like this because it describes fully and precisely the structure of an LSTM cell, in a way which mostly avoids ML/stats jargon. From this article I can correctly and easily read off the model architecture it describes, as a composition of smooth maps on modules over the real numbers, which is more than what one can say about a lot of papers.
- colah3 9y ago
  Hi! I'm the author. I was also pretty surprised by the amount of attention this article has gotten. That said, I can give you my take on how it advanced the dialogue. As you note, the main contribution of this article is the diagrams -- but I think it's a bit deeper than being pretty. Previously, diagrams of LSTMs had lots of loops in them. This made them hard to understand. (In fact, they were ambiguous because of unclear execution order.) The diagrams in this article unrolled the loops, which seems to be a much easier way to reason about LSTMs. They also work at the abstraction of layers instead of weights and matrix multiplies, which make the diagram less noisy and focuses the reader on the important ideas. Since diagrams were previously challenging to consume and understand, explanations prior to my article tended to focus on the six equations than define an LSTM. While that's a relatively clear way to think about LSTMs once you've really internalized them, it can be very hard when you are trying to understand them for the first time. In particular, the equations introduce ~15 variables. If you aren't already familiar with those variables, it's extremely challenging to keep them all in your working memory and understand what's going on. In contrast, the diagrams don't require a new reader to hold lots of things in working memory, and visually hint a great deal of the narrative (like the horizontal line at the top).
hinkley 9y ago
It takes him 12 paragraphs to define the acronym in the title.

rahulchhabra07 9y ago

Had been my first reading on LSTM and was so smooth and lucid. I did need to read more literature and code, but a great resource to begin with.

blueyes 9y ago

This is also a good one: https://deeplearning4j.org/lstm https://deeplearning4j.org/lstm

stared 9y ago

It's a great text, but I recommend supplementing it with: http://blog.echen.me/2017/05/30/exploring-lstms/ http://blog.echen.me/2017/05/30/exploring-lstms/. Additionally, GRU may be simpler to understand memory unit.

nafizh 9y ago

prev discussion: https://news.ycombinator.com/item?id=10130341 https://news.ycombinator.com/item?id=10130341