Yes, the state for both is a long window / sequence. Under this view, for the transformer we do not need to compute anything for the previous tokens as due to t

Yes, the state for both is a long window / sequence. Under this view, for the transformer we do not need to compute anything for the previous tokens as due to the causal nature of the model, the tokens at [0, ... N-1] are oblivious to the token N. For token N we can use the previous computations since they do not change.