6 ms·
Exploring LSTMs
- inlineint 9y agoI personally find recurrent highway networks (RHNs) as described in [1] to be easier to understand and remember the formulas for than the original LSTM. Because as they are generalizations of LSTM, if one understands RHNs, one can understand LSTMs as just a particular case of RHN. Instead of handwaving about "forgetting", it is IMO better to understand the problem of vanishing gradients and how can forget gates actually help with them. And Jürgen Schmidhuber, the inventor of LSTM, is a co-author of the RHN paper. [1] https://arxiv.org/abs/1607.03474 https://arxiv.org/abs/1607.03474
- make3 9y agoYes, but LSTMs are fucking everywhere in the industry so understanding them is crucial
- mrplank 9y agoLSTMs are on their retour in my opinion. They are a hack to make memory in recurrent networks more persistent. In practice they overfit too easy. They are being replaced with convolutional networks. Have a look at the latest paper from Facebook about translation for more details.
- Drdrdrq 9y agoThe way I see it, the difference is that with CNN you have fixed maximum timeframe in which knowledge about world is preserved, while LSTMs and RNNs in general do not impose such restrictions. This makes them better suited for some applications. If I am missing something please correct me.
- mrdrozdov 9y agoI think LSTMs are far from being replaced. They work too well and are too simple to use.
- visarga 9y agoLSTMs are both amazing and not quite good enough. They seem to be too complicated for what they do well, and not quite complex enough for what they can't do so well. The main limitation is that they mix structure with style, or type with value. For example, if you want an LSTM to learn addition, if you taught it to operate on numbers of 6 digits it won't be able to generalize on numbers of 20 digits. That's because it doesn't factorize the input into separate meaningful parts. The next step in LSTMs will be to operate over relational graphs so they only have to learn function and not structure at the same time. That way they will be able to generalize more between different situations and be much more useful. Graphs can be represented as adjacency matrices and data as vectors. By multiplying vector with matrix, you can do graph computation. Recurring graph computations are a lot like LSTMs. That's why I think LSTMs are going to become more invariant to permutation and object composition in the future, by using graph data representation instead of flat euclidean vectors, and typed data instead of untyped data. So they are going to become strongly typed, graph RNNs. With such toys we can do visual and text based reasoning, and physical simulation.
- LukeB42 9y agoAny links to implementations?
- deleted 9y ago[deleted]
- visarga 9y agoYes, one of the links is from DeepMind: "A simple neural network module for relational reasoning" https://arxiv.org/abs/1706.01427 https://arxiv.org/abs/1706.01427 Another one is from Thomas Kipf: "Graph Convolutional Matrix Completion" https://arxiv.org/abs/1706.02263 https://arxiv.org/abs/1706.02263
- LukeB42 9y agoThanks for elucidating the contents of the DeepMind paper and for hipping me to the Graph Convolutional Matrix Completion paper.
- deepnotderp 9y agoThis is really nice but did we really have to use Java as an example language?
- minimaxir 9y agoIs there code for the coloring of neurons per-character as in the post? I've seen that type of visualization on similar posts and am curious if there is a library for it. (the original char-rnn post [http://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://karpathy.github.io/2015/05/21/rnn-effectiveness/] indicates that it is custom Code/CSS/HTML)
- trwoway 9y agoFrom Karpathy's blog post : http://cs.stanford.edu/people/karpathy/viscode.zip http://cs.stanford.edu/people/karpathy/viscode.zip
- dirtyaura 9y agoReally great work on visualizing neurons! Is anyone working with LSTMs in a production setting? Any tips on what are the biggest challenges? Jeremy Howard said in fast.ai course that in the applied setting, simpler GRUs work much better and has replaced LSTMs. Comments about this?
- denzil_correa 9y ago> simpler GRUs What does GRU stand for?
- davik 9y agoGated recurrent unit, it has fewer gates than LSTMs usually.
- olavgg 9y agoOur chatbot uses LSTM(GRU) for prediction, we see about the same performance as LSTM, but training is more efficient.
- agibsonccc 9y agoYes the bulk of our business is time series. This includes everything from hardware break downs to fraud detection. I think Jeremy has some good points but in general, but I wouldn't assume that everything is binary. (By this, I mean look at these kinds of terse statements with a bit of nuance) Usually as long as you have a high amount of regularization and use truncated backprop through time in training you can learn some fairly useful classification and forecasting problems. Beyond that standard neural net tuning applies. Eg: normalize your data, pay attention to your weight initialization, understand what loss function you're using,..
- griffinkelly 9y agoSo if LSTMs are purposely forgetting, do you need less training data than a CNN?
- 9y ago
- natch 9y agoLSTM is "Long Short Term Memory," since the tutorial never mentions what it stands for. https://en.wikipedia.org/wiki/Long_short-term_memory https://en.wikipedia.org/wiki/Long_short-term_memory
- jeffjose 9y agoIt's defined few sections in, but yes, your larger point stands.
- raarts 9y agoCan someone provide a tl;dr ?
- avaer 9y agoNeural nets that update over time need a way to know what to keep and what to forget from last time. So let's learn what to keep and forget... by using more neural nets. Disclaimer: I just learned what an LSTM was. But it's a good article.
- backpropaganda 9y agoThis is a surprisingly accurate tl;dr for someone who just learnt what LSTMs are.
- dnautics 9y agoOne more key point:. Memory is additive, not multiplicative.
- SAI_Peregrinus 9y agoAnd LSTM stands for Long/Short Term Memory, the definition is only found about a third of the way through the article.
- thrw1001 9y agoKnowing what the actual units do, I prefer spelling out the acronym as Long Short-Term Memory. They implement short-term memory, like all RNNs do, with a slight improvement to make it long.
- Seanny123 9y agoIs the code for generating the reactions from the LSTM hidden units posted anywhere? That was the best part for me and I'd love to use it in my own projects.
- Joris1225 9y agoIn Andrej Karpathy's excellent blogpost on RNNs[1], he links to some of the code he used for visualisation[2]. I have done something similar. In general: put your the activations of each cell memory over time through tanh and color it based on that. [1] http://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://karpathy.github.io/2015/05/21/rnn-effectiveness/ [2] http://cs.stanford.edu/people/karpathy/viscode.zip http://cs.stanford.edu/people/karpathy/viscode.zip
- YeGoblynQueenne 9y agoIn the experiment on teaching an LSTM to count, it's useful to note that the examples it's trained on are derivations [1] from a grammar a^nb^n (with n > 0), a classic example of a Context-Freee Grammar (CFG). It's well understood that CFGs can not be induced from examples. Which accounts for the fact that LSTMs cannot learn "counting" in this manner, nor indeed can any other learning method that learns from examples. _______________ [1] "Strings generated from" [2] The same goes for any formal grammars other than finite ones, as in simpler than regular.
- MrQuincle 9y ago> It's well understood that CFGs can not be induced from examples I think you mean something more specific (e.g. polynomial in a particular sense). + Automatic Learning of Context-Free Grammar (Chen et al.): http://www.aclweb.org/anthology/O06-1004 http://www.aclweb.org/anthology/O06-1004 + Learning context-free grammars from structural data in polynomial time (Sakakibara, 1994): http://www.sciencedirect.com/science/article/pii/030439759090017C http://www.sciencedirect.com/science/article/pii/03043975909... (uses positive skeletons) Nice overview: http://staff.icar.cnr.it/staff/ruffolo/public_html/progetti/projects/07.Learning%20di%20SCFG/SCFG%20learning/index-c1.pdf http://staff.icar.cnr.it/staff/ruffolo/public_html/progetti/...
- avmich 9y agoAlso http://nbviewer.jupyter.org/url/norvig.com/ipython/xkcd1313.ipynb http://nbviewer.jupyter.org/url/norvig.com/ipython/xkcd1313.... , considering regexes are a subset of CFGs.
- YeGoblynQueenne 9y agoFrom a quick glance, the example doesn't quite learn regular grammars - rather, it reduces them to finite grammars (e.g. the finite grammar of all US presidential winners and losers) and learns (or, more accurately, invents) a regular expression for them. Finite grammar induction from positive examples only is feasible in polynomial time, so Peter Norvig's notebook will not cause the fabric of the space-time continuum to be torn asunder, I am sure.
- deleted 9y ago[deleted]
- CyberDildonics 9y ago> I once though LSTMs were tricky, but LSTMs are actually very easy ... You would think an article like this would define LSTM somewhere.
- mrplank 9y agoGoogle Brain outperforms LSTMs with Convolutional Networks in speed and accuracy, seeming to confirm LSTMs are not optimal for NLP at least: https://arxiv.org/pdf/1706.03762.pdf https://arxiv.org/pdf/1706.03762.pdf