>Likely there are much more efficient ways to train these models, it shouldn't be this hard for them to learn arithmetics. Oh really? And what would that look

>Likely there are much more efficient ways to train these models, it shouldn't be this hard for them to learn arithmetics. Oh really? And what would that look like? People have literally been looking at this for decades. And yet we're still using gradient descent with a few tweaks. So I would not bet on anyone coming up with something revolutionary. Especially not while hardware regularly revolutionizes the capabilities of models trained in this ancient way. Chipmakers have always allowed us to overcome the hurdle created by our lack of advances on the training architecture front. No matter whether you look at convnets or transformers.