Architectural innovations can definitely help with training/inference speed - that was the case with convolutional networks too - but for model performance (as

Architectural innovations can definitely help with training/inference speed - that was the case with convolutional networks too - but for model performance (as in 'intelligence' not speed) dataset size and quality were always more important. Even with classical ML models the advice was to stop tweaking the model and clean the data/gather more data first.