If it helps, you can think of a RNN as being analogous to a finite state machine. But instead of a single discrete state, it's a continuous, high-dimensional ve

If it helps, you can think of a RNN as being analogous to a finite state machine. But instead of a single discrete state, it's a continuous, high-dimensional vector. That has the extremely important effect that the output is a continuous function of the input, which is necessary for training using gradient descent.