They can technically be as long as you want them, but in practice there are still severe constraints. LSTMs alleviate the gradient problems, but you still get r

They can technically be as long as you want them, but in practice there are still severe constraints. LSTMs alleviate the gradient problems, but you still get real trouble with long-term dependencies. Alex Graves and some others in DeepMind have focused a lot in the past year or so on developing practical differentiable data structures, so that the LSTM can read and write to an external memory (and save its precious internal state for more immediate needs) yet still be trainable via backpropagation.