3 ms·
Wait, so there's a way to make a model as smart as GPT but with less parameters? Isn't that why it's so good?
by s1k3s 4y ago
Wait, so there's a way to make a model as smart as GPT but with less parameters? Isn't that why it's so good?
- endisneigh 4y agothat's a complicated question to answer. what I'd say is that more parameters makes the model more robust, but there are diminishing returns. optimizations are under way
- month13 4y agoThis is an older paper, but DeepMind alleges in their Chinchilla paper that far better performance can be extracted with fewer parameters; quote "We find that current large language models are significantly under-trained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant." It's difficult to evaluate a LLM's performance as it's all qualitative, but Meta's LLaMA has been doing quite well, at even 13B parameters.
- astrange 4y agoChinchilla is aimed at finding a cost-performance tradeoff as well, not the optimal amount of training. If cost is no barrier because it'll be used forever, then probably there's no amount of training that's good enough.
- monocasa 4y agoThe rumor I've heard is that GPT4 didn't meaningfully increase the parameter count versus GPT3.5, but instead focused on training and structural improvements.
- jazzyjackson 4y agoYou can train a small model to behave like the large model at a subset of tasks.