4 ms·
I will hazard a guess as to why this might be better than LSTM. LSTM maintains a vector that represents the long short term memory, but once information is cle
by andrewtbham 11y ago
I will hazard a guess as to why this might be better than LSTM. LSTM maintains a vector that represents the long short term memory, but once information is cleared from the vector, there is no way to get it back... whereas a stack you can keep going further down the stack if needed.
http://colah.github.io/posts/2015-08-Understanding-LSTMs/ http://colah.github.io/posts/2015-08-Understanding-LSTMs/
- sxyuan 11y agoOne related work that came to mind was Neural Turing Machines, which augments the LSTM network with a (fixed size) memory [1]. I did see a brief reference to the Graves et al. paper in the paper linked to in the original post, but I'm also wondering if someone more knowledgeable could elaborate on the difference. My own guess would be that the models are similar (both using NNs with external memory, trained through backprop), but this has a simpler memory structure and so could be trained more easily. It seems like the choice of regular RNNs vs. an LSTM network doesn't matter much since the Graves paper also compared LSTM vs. feedforward controllers and the results were about the same, with the feedforward controllers actually learning a bit faster. The key of both works seems to be in making a continuous system that can be trained to use external memory through backprop. [1] http://arxiv.org/abs/1410.5401 http://arxiv.org/abs/1410.5401