I'm a strong opponent of just how large Large Language Models have become. Yeah, they can do some cool stuff, but I really, really think >100B parameters is a r

I'm a strong opponent of just how large Large Language Models have become. Yeah, they can do some cool stuff, but I really, really think >100B parameters is a rather silly number. In my opinion, the way you organize and train the parameters is far more important than the parameter count, which I think TFA supports pretty solidly. While I've never personally overseen the training of a model bigger than 2B params, I have trained a lot of smaller models and they consistently surprise me in how capable they are when trained in clever ways on adequately diverse datasets. My wild, unfounded speculation? AGI in 24GB VRAM is feasible :P