4 ms·
I was not referring to typical RNNs, but LSTMs or RNNs with 'attention'. They are designed to overcome vanishing/ exploding gradient problems and hold arbitrary
by orthoganol 10y ago
I was not referring to typical RNNs, but LSTMs or RNNs with 'attention'. They are designed to overcome vanishing/ exploding gradient problems and hold arbitrary memory lengths.
- vintermann 10y agoThey can technically be as long as you want them, but in practice there are still severe constraints. LSTMs alleviate the gradient problems, but you still get real trouble with long-term dependencies. Alex Graves and some others in DeepMind have focused a lot in the past year or so on developing practical differentiable data structures, so that the LSTM can read and write to an external memory (and save its precious internal state for more immediate needs) yet still be trainable via backpropagation.
- modeless 10y agoThe memory provided by an attention architecture is immutable and has only one addressing mode. The memory in this paper is mutable (hence "Neural Turing Machine") and has several different addressing modes.