4 ms·
I'm curious - what happens if one prunes the halved model again (if that's possible with the same method), would it start losing accuracy?
by drdaeman 2y ago
I'm curious - what happens if one prunes the halved model again (if that's possible with the same method), would it start losing accuracy?
- koolba 2y agoLet’s take it a step further and accept some inaccuracy. If we apply the Pareto principle[1], we should get 80% of the accuracy for 20% of the size. Compounding that four times, we should get .8^4 = 40% of the accuracy for .2^4 = .16% of the size. That’d be about 1 GB for the current largest model. [1]: https://en.wikipedia.org/wiki/Pareto_principle https://en.wikipedia.org/wiki/Pareto_principle
- regularfry 2y agoNo need to just use 80/20 as the split. The article says (on one benchmark) you get 97.3% of the accuracy for 50% of the size. So blindly applying Pareto you get (compounding nine times, because why not) 78% of the accuracy for 0.2% of the size. Something tells me that's a little optimistic.
- SubiculumCode 2y agoI was thinking the same. On HF, I see 4bit gguf of this 2:4 model, and I'm like...that works?
- dcreater 2y agoLink?
- SubiculumCode 2y agohttps://huggingface.co/QuantFactory/Sparse-Llama-3.1-8B-2of4-GGUF https://huggingface.co/QuantFactory/Sparse-Llama-3.1-8B-2of4...
- bob1029 2y agoAt some point you will hit the interpolation threshold and your model will overfit perfectly to the training set. The gargantuan # of parameters is what buys you the generalization properties that everyone is interested in. A very reduced model may still look & sound competent on the surface, but extensive use by domain experts would quickly highlight the cost of this.