3 ms·

A lot of weird behaviors and problems of LLMs trace back to tokenization

30 points by stared 3y ago

dang 3y ago
Related ongoing thread: Let's Build the GPT Tokenizer [video] - https://news.ycombinator.com/item?id=39443965 https://news.ycombinator.com/item?id=39443965 - Feb 2024 (12 comments)
stared 3y ago
Tokenization is a crude text compression, which happens at the cost of obfuscating the data. Without it, we would effectively reduce the context size and increase the training time. For OpenAI tokenizer - by a factor of 4. Is there a workaround, or are there only trade-offs?
h2odragon 3y ago
I've said it for years; the pre-filtering of data is just putting blinders on before you even begin to train. I have rarely managed to make myself understood. real models should be able to infer "HTML as she is spoken" and separate it from the human languages it complements. what we got is "models of interpolated views of language"
- Arnt 3y ago
  IYUM that you've asserted it for years without argument, but rarely managed to persuade anyone. Try arguing with arguments instead of asserting. Something like "Tokenization superficially appears to be a boost just like lexers appear to help parsers on first glance, but on closer reflection it's a net negative because [insert arguments here]".
ThrowawayTestr 3y ago
Can we make the size of tokens arbitrary and make them the size of words?
- theGnuMe 3y ago
  Yes but how would that help?
xg15 3y ago
Wasn't one of the important properties of tokens that they allow the LLM to represent a sort of "semantic unit"? I.e. If you generate an embedding over words, the embedding can capture meaningful semantic relations between words, because most words do have a relatively narrowly-defined meaning. In contrast, an embedding over letters/characters/bytes/etc can't do the same because the meaning of those depends entirely on context. I thought, tokens were sort of a compromise between the two: they still allowing meaningful embeddings for frequent, "large" tokens while also including "small" tokens that can be used to make up new words.
- deleted 3y ago
  [deleted]
mike_hearn 3y ago
Tokenization is my pet peeve with LLMs as I suspect it makes it much harder for them to learn coding, but it's difficult to do anything about because doing better requires you to train a model from scratch. In turn that's so expensive that it hardly happens and the people who do it tend to focus on higher priorities, leaving tokenization alone. I guess at some point we'll see some progress here. Things I'd like to see tweaked: • All code being reformatted as part of the cleaning process to use a canonical style designed for LLMs. For example, reindenting to use tabs instead of spaces, unwrapping lines. Way too many tokens in the GPT4 token list have random bits of whitespace prepended to them meaning they can only be used in certain nesting levels, which must make it harder for the model to learn. • Forcing language keywords to be a single token. GPT4 has tokens like " public", " public", " Public" etc. • Special tokens being used to indicate start/end of identifiers, then decompounding and decasing words during tokenization, i.e. "void flipTheSwitch(" -> "void <id-start> flip the switch <id-end> (". This avoids tokens that are compound words which forces the model to relearn their meaning from scratch. The renderer would then need to do the reverse during inferencing. • Decompounding dereference operators like "." or "->". The GPT4 token list is full of tokens like ".square", ".attribute" etc. • Clearing away all the tokens that are just punctuation compounds like "))));\n" The above changes would slow down inference by requiring more tokens to be generated, and would require renderers to be taught how to reformat code on the fly to a form better suited for humans (e.g. shorter lines), but intuitively should make it far easier for smaller models to learn coding and that in turn might make things faster again. More speculative ideas include: • Reserve some special tokens for bracketing IDE-style features like inlining invisible JavaDocs, so inference can be rolled forward and backwards when the logprob distribution is too flat. • Ditto for inferred types if the cleaning process can be tightly integrated with an IDE or language server (e.g. "val a = b.c.d()" -> "val a /String/ = /Foo/b./Bar/c./String/d()") There's just so much that can be done here, the amount of low hanging fruit is enormous. But right now I suspect the industry is stuck in a kind of local minima where if you're willing and able to train an LLM from scratch you also aren't terribly price sensitive and would rather bitter-lesson your way to better results, or focus on core algorithmic improvements that affect more than just coding, rather than spend time improving tokenization schemes. I also suspect that the Python focus of the AI community means they just aren't thinking very hard about ways to help LLMs with strongly typed languages.

30 points by stared 3y ago

dang 3y ago

Related ongoing thread: Let's Build the GPT Tokenizer [video] - https://news.ycombinator.com/item?id=39443965 https://news.ycombinator.com/item?id=39443965 - Feb 2024 (12 comments)

stared 3y ago

Tokenization is a crude text compression, which happens at the cost of obfuscating the data. Without it, we would effectively reduce the context size and increase the training time. For OpenAI tokenizer - by a factor of 4. Is there a workaround, or are there only trade-offs?

h2odragon 3y ago

I've said it for years; the pre-filtering of data is just putting blinders on before you even begin to train. I have rarely managed to make myself understood. real models should be able to infer "HTML as she is spoken" and separate it from the human languages it complements. what we got is "models of interpolated views of language"

Arnt 3y ago
IYUM that you've asserted it for years without argument, but rarely managed to persuade anyone. Try arguing with arguments instead of asserting. Something like "Tokenization superficially appears to be a boost just like lexers appear to help parsers on first glance, but on closer reflection it's a net negative because [insert arguments here]".

ThrowawayTestr 3y ago

Can we make the size of tokens arbitrary and make them the size of words?

theGnuMe 3y ago
Yes but how would that help?

xg15 3y ago

Wasn't one of the important properties of tokens that they allow the LLM to represent a sort of "semantic unit"? I.e. If you generate an embedding over words, the embedding can capture meaningful semantic relations between words, because most words do have a relatively narrowly-defined meaning. In contrast, an embedding over letters/characters/bytes/etc can't do the same because the meaning of those depends entirely on context. I thought, tokens were sort of a compromise between the two: they still allowing meaningful embeddings for frequent, "large" tokens while also including "small" tokens that can be used to make up new words.

deleted 3y ago
[deleted]

mike_hearn 3y ago

Tokenization is my pet peeve with LLMs as I suspect it makes it much harder for them to learn coding, but it's difficult to do anything about because doing better requires you to train a model from scratch. In turn that's so expensive that it hardly happens and the people who do it tend to focus on higher priorities, leaving tokenization alone. I guess at some point we'll see some progress here. Things I'd like to see tweaked: • All code being reformatted as part of the cleaning process to use a canonical style designed for LLMs. For example, reindenting to use tabs instead of spaces, unwrapping lines. Way too many tokens in the GPT4 token list have random bits of whitespace prepended to them meaning they can only be used in certain nesting levels, which must make it harder for the model to learn. • Forcing language keywords to be a single token. GPT4 has tokens like " public", " public", " Public" etc. • Special tokens being used to indicate start/end of identifiers, then decompounding and decasing words during tokenization, i.e. "void flipTheSwitch(" -> "void <id-start> flip the switch <id-end> (". This avoids tokens that are compound words which forces the model to relearn their meaning from scratch. The renderer would then need to do the reverse during inferencing. • Decompounding dereference operators like "." or "->". The GPT4 token list is full of tokens like ".square", ".attribute" etc. • Clearing away all the tokens that are just punctuation compounds like "))));\n" The above changes would slow down inference by requiring more tokens to be generated, and would require renderers to be taught how to reformat code on the fly to a form better suited for humans (e.g. shorter lines), but intuitively should make it far easier for smaller models to learn coding and that in turn might make things faster again. More speculative ideas include: • Reserve some special tokens for bracketing IDE-style features like inlining invisible JavaDocs, so inference can be rolled forward and backwards when the logprob distribution is too flat. • Ditto for inferred types if the cleaning process can be tightly integrated with an IDE or language server (e.g. "val a = b.c.d()" -> "val a /String/ = /Foo/b./Bar/c./String/d()") There's just so much that can be done here, the amount of low hanging fruit is enormous. But right now I suspect the industry is stuck in a kind of local minima where if you're willing and able to train an LLM from scratch you also aren't terribly price sensitive and would rather bitter-lesson your way to better results, or focus on core algorithmic improvements that affect more than just coding, rather than spend time improving tokenization schemes. I also suspect that the Python focus of the AI community means they just aren't thinking very hard about ways to help LLMs with strongly typed languages.