Not sure if this is a good introduction, but a good second paper to read is https://arxiv.org/abs/2106.06981 https://arxiv.org/abs/2106.06981 You can think of

Not sure if this is a good introduction, but a good second paper to read is https://arxiv.org/abs/2106.06981 https://arxiv.org/abs/2106.06981 You can think of finite state machines as being two functions: f(input, state) = output, and g(input, state) = next_state. (Traditional FSMs have 3 ‘output’ states, basically ‘terminated - success’, ‘terminated - failure’, and ‘still working’, but in theory it makes sense to fully generalize it). If you think about plain neural networks as approximating arbitrary functions f(input) = output, then recurrent neural networks are “continuous state machines”, where you have the same two functions f(input, state) = output, and g(input, state) = next_state, except instead of being finite symbols, they’re continuous points in N dimensional space. This, at least to me, clarifies why recurrent neural networks work on simple and short time-based problems, but can’t efficiently generalize to complex problems—they’re just FSMs! The paper I linked above provides a similar high-level computational analogy to how transformers work.