3 ms·
I wonder how far we can take this. Is 1B parameters theoretically "expressive" enough for GPT-4 like performance? I wonder how far off "theoretically optimal" w
by theLiminator 3y ago
I wonder how far we can take this. Is 1B parameters theoretically "expressive" enough for GPT-4 like performance? I wonder how far off "theoretically optimal" we are in terms of performance/parameters ratio.
- dontwearitout 3y agoGood open questions. I suspect we'll see models distilled and compressed down to retain most of their "common sense", "core knowledge", and reasoning ability, while stripping out the gigabytes of random trivia most people will never need.