3 ms·
Describing LLMs: "Training data: 1 to 2 trillion tokens" Is number of tokens a good metric, given relationships between tokens is what's important? An LLM wit
by coldtea 4y ago
Describing LLMs: "Training data: 1 to 2 trillion tokens"
Is number of tokens a good metric, given relationships between tokens is what's important?
An LLM with 100000 trillion lexically shorted tokens, given one by one, wont be able to do anything except perhaps spell checking.
I guess the idea is that tokens are given in such "regular" forms (books, posts, webpages) that their mere count is a good proxy for number of relevant relationships.
- eternalban 4y agoI think the underlying assumption is that there is (an invariant) structure and distribution that becomes more resolved (q) as more data is incorporated. Training LLMs also involves sequences of tokens by design so that is also implicit. The only venue left is to question whether this (n -> q) is a linear relationship or something else. That would only matter if we were looking at absolute measure (to compare say an LLM architecture to something else), but as a comparative measure for LLMs it works, whether it is o(n) or o(logn) or whatever, since more tokens means more sequences and more sequences mean greater ability.