There's one important caveat, though I agree with your thrust: at GPT-3 scale, cutting params in half is a nontrivial optimization. So it's worth keeping an eye

There's one important caveat, though I agree with your thrust: at GPT-3 scale, cutting params in half is a nontrivial optimization. So it's worth keeping an eye out for that concern. (Yeah, none of us are anywhere near GPT-3 scale. But I spend most of my time thinking about scaling issues, and it was interesting to see your comment pop up; I would've agreed entirely with it myself, if not for seeing all the anguish caused by attempting to train and deploy billions of parameters.)