HMMs are much more similar to S3, S4, and other deep state-space models than to transformers. In fact, HMMs are discrete shallow state-space models. I believe

HMMs are much more similar to S3, S4, and other deep state-space models than to transformers. In fact, HMMs are discrete shallow state-space models. I believe S4 and successors might become serious contenders to transformers. For a detailed tutorial, see https://srush.github.io/annotated-s4 https://srush.github.io/annotated-s4