"LLMs are Markov Models"... There's a reason we don't use Markov models anymore though (for language modeling - still used heavily in RL). The density of infor

"LLMs are Markov Models"... There's a reason we don't use Markov models anymore though (for language modeling - still used heavily in RL). The density of information and composability you get from MHA makes the problem far more tractable.