>What exactly makes you so confident? Existing research? If you want something that scales as well as transformers you have to make the divergences I was talki

>What exactly makes you so confident? Existing research? If you want something that scales as well as transformers you have to make the divergences I was talking about. If you don't then it scales a lot worse. The Newton methods don't match transformer efficiency at scale. That's just a fact. >The world is not just labs that can afford billion dollar datacentres and selling access to SOTA LLMs at $30/Mtokens. Billion dollar labs want to save money too. If Modern RNNs were a massive unanimous win, they and everyone else would switch in a heartbeat, just like they did for transformers. The reason they don't is because these architectures at best simply match transformers, while introducing their own architectural issues.