I believe the problem is the training algorithm, backprop, not the model (NNs). "unstable fixed point attractor of Backpropagation through Time", as you said.

I believe the problem is the training algorithm, backprop, not the model (NNs). "unstable fixed point attractor of Backpropagation through Time", as you said. Ilya Sutskever in Geoff Hinton's lab has had great success recently using hessian-free optimization to train recurrent networks. He has trained character-level RNNs on Wikipedia, and they can generate very long sequences of quasi-plausible text. In particular, it seems like they can remember many symbols. See his most recent pubs: http://www.cs.toronto.edu/~ilya/pubs/ http://www.cs.toronto.edu/~ilya/pubs/