13 ms·
How would this be any different from running one of the lower parameter models?
by bick_nyers 3y ago
How would this be any different from running one of the lower parameter models?
- sva_ 3y agoIt says in the abstract > at minimal loss of accuracy Suggesting that there is a lot of redundancy in the weights.
- theLiminator 3y agoI wonder how far we can take this. Is 1B parameters theoretically "expressive" enough for GPT-4 like performance? I wonder how far off "theoretically optimal" we are in terms of performance/parameters ratio.
- dontwearitout 3y agoGood open questions. I suspect we'll see models distilled and compressed down to retain most of their "common sense", "core knowledge", and reasoning ability, while stripping out the gigabytes of random trivia most people will never need.
- amelius 3y agoThis makes me wonder what would happen if you took the sparse model, reset its weights, then trained it with the original training data.
- sva_ 3y agoVery interesting idea. I'd hypothesize that it won't achieve the same(ish) accuracy, and that pruning might be required (similar to how humans go through a heavy pruning phase at an early age[0]). Would be worth setting up an experiment on a smaller scale. As some other commentator stated, there's currently a lot of low hanging fruit in optimizing NN. 0. https://en.m.wikipedia.org/wiki/Synaptic_pruning https://en.m.wikipedia.org/wiki/Synaptic_pruning
- avereveard 3y agoLarger models seem to handle much better introspection, makes for better backend for sourced knowledge extraction