4 ms·

Could anyone who's an expert comment why there seems to be such a focus on discussing tokenizers? It seems every other day there's a new article or implementati

Could anyone who's an expert comment why there seems to be such a focus on discussing tokenizers? It seems every other day there's a new article or implementation of a tokenizer on HN. But downstream from that, rarely anything. As a non-expert I would have thought to tokenizing is just one step.

simonw 3y ago
I just think they're interesting. From a practical point of view they only really matter in that we have to think carefully about how to use our token budget.
ftxbro 3y ago
The reason it's trending today is because of the phenomenon of Glitch Tokens. They thought all Glitch Tokens had been removed by GPT-4 but apparently one is still left. If you go down the rabbit hole on Glitch Tokens it gets ... really really weird.
- kenjackson 3y ago
  But does the tokenizer have anything to do with Glitch Tokens? Glitch Tokens seem more like a function of the neural network. I'm saying this with only a surface level understanding of glitch tokens.
  - GeneralMayhem 3y ago
    It does a bit, because the fact that they're able to persist is sort of an artifact of how naive the tokenizer is (it's a counting operation based on n-grams), and that it runs as a separate step. There's no feedback from the transformer to the tokenizer to say "hey, this token is actually pretty meaningless, maybe try again on that one". That means that strings of characters that are common but very low semantic value, like the example of Reddit usernames that mostly post on /r/counting, will be included in the model's vocabulary even though they're not interesting. When humans see extremely low-information-density data, we can forget it. And the model can too, but only kind of - it can forget (or rather, never learn) what the "word" means, but it can't forget that it's a word.
deleted 3y ago
[deleted]
SkyPuncher 3y ago
Tokens are the primitives that most LLMs (and broadly a lot of NLP) works with. While, you and I would expect whole-words to be tokens, many tokens are shorter - 3 to 4 characters - and don't always match the sentence structure you and I expect. This can create some interesting challenges and unexpected behavior. It also makes certain things, like vectorization, a challenge since tokens may not map 1:1 with the words you intend to weight them against.
- whimsicalism 3y ago
  You are using the word vectorization in an idiosyncratic way, you are referring to the process of embedding words?
  - SkyPuncher 3y ago
    Because I’m not an expert in this area. I know it well enough to build products around it, but it’s not my deep area of expertise. Just trying to provide an example.
- hsjqllzlfkf 3y ago
  Your answer explains what tokenizers are, which isn't what I asked. You also told me something interesting about tokenizers, which is also not what I asked. Can you tell me anything NOT about tokenized? This is my point.
  - mike_hearn 3y ago
    The reason it's not discussed much is that what goes on downstream of tokenization is extremely opaque. It's lots of layers of the transformer network so the overall structure is documented but what exactly those numbers mean is hard to figure out. There's an article here where the structure of an image generation network is explored a bit: https://openai.com/research/sparse-transformer https://openai.com/research/sparse-transformer They have a visualization of what the different layers are paying attention to. There are also some good explanations of transformers elsewhere online. This one is old but I found it helpful: http://jalammar.github.io/illustrated-transformer/ http://jalammar.github.io/illustrated-transformer/
Der_Einzige 3y ago
Most of the shitty behavior of LLMs on syntactic and lexical tasks are due to the tokenizer and not due to the LLM itself. Having even tiny changes in tokenization has massive downstream effects on LLM behavior.
llm_nerd 3y ago
Tokenizing is just one very trivial step, and it is probably the simplest and least interesting part of the process. Embedding vectors are dramatically more interesting and actually useful. There is a mad rush to write articles in the LLM / ML / AI space to show that you haven't been left behind (like a FOMO, but more a FO-looking-like-you-MO). Tokenizers are by far the easiest part of that stack to grok, so the end result are a seemingly infinite selection of tokenization submissions.

simonw 3y ago

I just think they're interesting. From a practical point of view they only really matter in that we have to think carefully about how to use our token budget.

ftxbro 3y ago

The reason it's trending today is because of the phenomenon of Glitch Tokens. They thought all Glitch Tokens had been removed by GPT-4 but apparently one is still left. If you go down the rabbit hole on Glitch Tokens it gets ... really really weird.

kenjackson 3y ago
But does the tokenizer have anything to do with Glitch Tokens? Glitch Tokens seem more like a function of the neural network. I'm saying this with only a surface level understanding of glitch tokens.
- GeneralMayhem 3y ago
  It does a bit, because the fact that they're able to persist is sort of an artifact of how naive the tokenizer is (it's a counting operation based on n-grams), and that it runs as a separate step. There's no feedback from the transformer to the tokenizer to say "hey, this token is actually pretty meaningless, maybe try again on that one". That means that strings of characters that are common but very low semantic value, like the example of Reddit usernames that mostly post on /r/counting, will be included in the model's vocabulary even though they're not interesting. When humans see extremely low-information-density data, we can forget it. And the model can too, but only kind of - it can forget (or rather, never learn) what the "word" means, but it can't forget that it's a word.

deleted 3y ago

[deleted]

SkyPuncher 3y ago

Tokens are the primitives that most LLMs (and broadly a lot of NLP) works with. While, you and I would expect whole-words to be tokens, many tokens are shorter - 3 to 4 characters - and don't always match the sentence structure you and I expect. This can create some interesting challenges and unexpected behavior. It also makes certain things, like vectorization, a challenge since tokens may not map 1:1 with the words you intend to weight them against.

whimsicalism 3y ago
You are using the word vectorization in an idiosyncratic way, you are referring to the process of embedding words?
- SkyPuncher 3y ago
  Because I’m not an expert in this area. I know it well enough to build products around it, but it’s not my deep area of expertise. Just trying to provide an example.
hsjqllzlfkf 3y ago
Your answer explains what tokenizers are, which isn't what I asked. You also told me something interesting about tokenizers, which is also not what I asked. Can you tell me anything NOT about tokenized? This is my point.
- mike_hearn 3y ago
  The reason it's not discussed much is that what goes on downstream of tokenization is extremely opaque. It's lots of layers of the transformer network so the overall structure is documented but what exactly those numbers mean is hard to figure out. There's an article here where the structure of an image generation network is explored a bit: https://openai.com/research/sparse-transformer https://openai.com/research/sparse-transformer They have a visualization of what the different layers are paying attention to. There are also some good explanations of transformers elsewhere online. This one is old but I found it helpful: http://jalammar.github.io/illustrated-transformer/ http://jalammar.github.io/illustrated-transformer/

Der_Einzige 3y ago

Most of the shitty behavior of LLMs on syntactic and lexical tasks are due to the tokenizer and not due to the LLM itself. Having even tiny changes in tokenization has massive downstream effects on LLM behavior.

llm_nerd 3y ago

Tokenizing is just one very trivial step, and it is probably the simplest and least interesting part of the process. Embedding vectors are dramatically more interesting and actually useful. There is a mad rush to write articles in the LLM / ML / AI space to show that you haven't been left behind (like a FOMO, but more a FO-looking-like-you-MO). Tokenizers are by far the easiest part of that stack to grok, so the end result are a seemingly infinite selection of tokenization submissions.