I disagree with the editorialized abstract. State is not an illusion. There is a limit to what stateful LTI systems are capable of modeling. It is very interest

I disagree with the editorialized abstract. State is not an illusion. There is a limit to what stateful LTI systems are capable of modeling. It is very interesting that they've been able to prove what that limit is in the context of computation. From my reading, "non gated SSMs" are LTI systems. They're linear (the paper ignores the activation function) and the "does not depend on input" criteria is the controls way of saying "time invariant" (replace time with the domain of the input vector). It is not surprising that a system that cannot adapt to changing input conditions (by construction) is fundamentally limited to the problems it can model. This is why adaptive filters are studied - LTI systems are useful computationally, but limited because they cannot adapt. Relaxing the "TI" constraint greatly expands the domains of problems that can be solved. When the feedback matrix is diagonal you have an FIR system. While "finite impulse response" has as specific mathematical definition, conceptually it means the state at time n depends solely on past inputs and states n - N where N is the size of the state vector. So of course, if the feedback matrix is diagonal, the system is limited in the kinds of stateful problems it can handle. What this paper is missing is that connection to the limits of LTI and connection to TC0, which is very interesting indeed. In other words, the state of the network is still infinitely long and definitely not an illusion. However if the problem requires the network to adapt to its input then a non-gated SSM is not going to be sufficient. That's an interesting research space.