4 ms·
It says in the abstract > at minimal loss of accuracy Suggesting that there is a lot of redundancy in the weights.
by sva_ 4y ago
It says in the abstract
> at minimal loss of accuracy
Suggesting that there is a lot of redundancy in the weights.
- theLiminator 4y agoI wonder how far we can take this. Is 1B parameters theoretically "expressive" enough for GPT-4 like performance? I wonder how far off "theoretically optimal" we are in terms of performance/parameters ratio.
- dontwearitout 4y agoGood open questions. I suspect we'll see models distilled and compressed down to retain most of their "common sense", "core knowledge", and reasoning ability, while stripping out the gigabytes of random trivia most people will never need.
- amelius 4y agoThis makes me wonder what would happen if you took the sparse model, reset its weights, then trained it with the original training data.
- sva_ 4y agoVery interesting idea. I'd hypothesize that it won't achieve the same(ish) accuracy, and that pruning might be required (similar to how humans go through a heavy pruning phase at an early age[0]). Would be worth setting up an experiment on a smaller scale. As some other commentator stated, there's currently a lot of low hanging fruit in optimizing NN. 0. https://en.m.wikipedia.org/wiki/Synaptic_pruning https://en.m.wikipedia.org/wiki/Synaptic_pruning