Yeah I mean, great for running inference (and obviously training smaller models). The problem is that Scaling Laws for Transformers show us that scaling model p

Yeah I mean, great for running inference (and obviously training smaller models). The problem is that Scaling Laws for Transformers show us that scaling model parameter count _up_, is a sure way to improve the model's performance over your task. So when you're playing nation-state hard-ball, since the architectures are _somewhat_ trivially copied upon being published, that means the best checkpoint goes to the one with the fastest GPU's. If China wasn't behind in deep learning, they will be.