3 ms·
> Most modern neural nets, and certainly LLMs, are overparameterized, which means they are completely capable of regurgitating exact training instances. No the
by Animats 3y ago
> Most modern neural nets, and certainly LLMs, are overparameterized, which means they are completely capable of regurgitating exact training instances.
No they're not.
GPT-3: Model size: 175 GB, training set size 300 billion tokens.
GPT-4: Not disclosed.
LLaMA: Model size: 65GB, training set size, 1 trillion tokens.
These things condense the data. They don't just index it, like a search engine.
[1] https://lambdalabs.com/blog/demystifying-gpt-3 https://lambdalabs.com/blog/demystifying-gpt-3
- WithinReason 3y agoAlso, having the same number of parameters as the training set doesn't mean you can memorize it