5 ms·
I can believe that you can get good performance on 14 NLP tasks after pruning 70% of the model. But for things like chatgpt the use cases are way more diverse a
by changoplatanero 3y ago
I can believe that you can get good performance on 14 NLP tasks after pruning 70% of the model. But for things like chatgpt the use cases are way more diverse and you can’t keep high performance for everything when you prune that many of the weights.
- lolinder 3y agoYou don't need to get good performance on everything with a single model—Mixture of Experts models can outperform equivalently sized monolithic models. My understanding is that GPT-4 is structured this way. If we can trim a model in different ways to get different specializations, that could be really effective.
- novaRom 3y agoI just tried Mistral MoE 8x7B model and it works a bit faster than llama-2-70B but it looks it has almost the same skills. In fact, all latest models of 13B-70B size are quite similar. Could it be large part of their training data is the same?
- cyanydeez 3y agothere might be a laws of average thing going on? Central Limit Theorem? there's a finite number of relevant language tokens. the trick of current LLM is finding what's basically the center of a vast series of probability.
- sigmoid10 3y ago>But for things like chatgpt the use cases are way more diverse and you can’t keep high performance for everything when you prune that many of the weights. This also has become increasingly obvious from recent developments in the field. Today, we regulalry see new models that come at a fraction of the size of GPT-3 and yet easily outperform it, especially on certain downstream tasks when fine-tuned correctly. These small models also retain some generality, but not as much as the really high end really large models like GPT-4. I'd say a sub 10B parameter model equal or better than GPT-3 overall is achievable, but not for GPT-4. At least not with current technology. However, that would still imply that it's possible to reduce parameter counts in common approaches by 95%. I'm pretty sure in a few years people will look back and smirk at the crude methods we used to train LLMs today.
- Der_Einzige 3y agoCorrect, people in the model optimization world will claim all day that it only marginally impacts performance, but I’ve extensively played with quantization and pruning methods, and can report that they do cripple models in ways that the authors either didn’t notice or on purpose omitted from their SOTA benchmark chasing. Most claims of models being even equal to GPT-3.5 are also significantly overblown. I haven’t seen one yet below 70 billion parameters which even comes close. Nothing is free in this world.