It depends, actually... The data and train time requirements seen to increase exponentially for linear gains in performance. As a result, you can often trade a

It depends, actually... The data and train time requirements seen to increase exponentially for linear gains in performance. As a result, you can often trade a 10x reduction in training time to get a model with 90+% of the real deal. And as we accumulate more architecture and efficiency tricks, the ceiling in what you can do locally goes up commensurately. There's also a whole world of data curation to improve training, which is likely to be great for small models and seems still underexplored.