2 ms·
That seems suboptimal, for the same reason LLMs are trained on tokens, and not characters. Tokens seem like a much better "unit of meaning" than characters. Cu
by TeMPOraL 3y ago
That seems suboptimal, for the same reason LLMs are trained on tokens, and not characters. Tokens seem like a much better "unit of meaning" than characters.
Curiously, this also applies to humans: we learn words first, spelling later; we almost always think in terms of words, subwords and phrases, and we're really fast at it - while anything to do with spelling seems to demand much more focus, and some degree of conscious hand-holding.
As for encoding sequence, I'm curious how that happens and need to find relevant papers, but I imagine there might be some "n-gram dimensions" in the vector, as in one value for "I'm a first token in a sequence", one value for "I'm a second token in a sequence", etc., which would encode the "occurs before"/"occurs after" relationships using few dimensions, leaving the rest for more interesting relationships.