>But for things like chatgpt the use cases are way more diverse and you can’t keep high performance for everything when you prune that many of the weights. Thi

>But for things like chatgpt the use cases are way more diverse and you can’t keep high performance for everything when you prune that many of the weights. This also has become increasingly obvious from recent developments in the field. Today, we regulalry see new models that come at a fraction of the size of GPT-3 and yet easily outperform it, especially on certain downstream tasks when fine-tuned correctly. These small models also retain some generality, but not as much as the really high end really large models like GPT-4. I'd say a sub 10B parameter model equal or better than GPT-3 overall is achievable, but not for GPT-4. At least not with current technology. However, that would still imply that it's possible to reduce parameter counts in common approaches by 95%. I'm pretty sure in a few years people will look back and smirk at the crude methods we used to train LLMs today.