There are many attempts at using attention-less approaches to the architecture, and Markov Chains being one of them. Nothing to compete with transformers yet th

There are many attempts at using attention-less approaches to the architecture, and Markov Chains being one of them. Nothing to compete with transformers yet though, AFAIK. Some people experiment with graph-based rules systems, like very smartly compressed grammars (otherwise the task is NP-Hard i think).