3 ms·

Your answer explains what tokenizers are, which isn't what I asked. You also told me something interesting about tokenizers, which is also not what I asked. Can

Your answer explains what tokenizers are, which isn't what I asked. You also told me something interesting about tokenizers, which is also not what I asked. Can you tell me anything NOT about tokenized? This is my point.

mike_hearn 3y ago
The reason it's not discussed much is that what goes on downstream of tokenization is extremely opaque. It's lots of layers of the transformer network so the overall structure is documented but what exactly those numbers mean is hard to figure out. There's an article here where the structure of an image generation network is explored a bit: https://openai.com/research/sparse-transformer https://openai.com/research/sparse-transformer They have a visualization of what the different layers are paying attention to. There are also some good explanations of transformers elsewhere online. This one is old but I found it helpful: http://jalammar.github.io/illustrated-transformer/ http://jalammar.github.io/illustrated-transformer/
- hsjqllzlfkf 3y ago
  This was my suspicion, thank you.
SkyPuncher 3y ago
With all due respect, this feels like asking me to talk about math without talking about numbers. Tokens are so closely tied to modern LLMs that’s it’s basically impossible to not talk about them. They’re getting a lot of attention because they are the primitive. They’re the thing of most interest for improving performance.
wolfgang42 3y ago
You asked why there was a focus on discussing tokenizers, and got an answer explaining why tokenizers are something people would want to discuss.
- washadjeffmad 3y ago
  > ...It seems every other day there's a new article or implementation of a tokenizer on HN. But downstream from that, rarely anything. As a non-expert I would have thought to tokenizing is just one step. If someone points out a preponderance of information on one step relative to all other steps, they probably are not asking for even more information about that step. People like to chip in with what they've recently learned, so one answer is that most people on HN don't understand much beyond the input layer. A better answer is that the relative complexity of the processes in subsequent layers increases substantially, along with the requisite background to understand them. They also don't share the relative commonality of the input layer, so fewer people are qualified to discuss them with any authority. That's where I am, so I get it. I'm working on building learning resources for a symposium, and it feels very much like "Step 1: Tokenize, Step 2: ???, Step 3: Output!".

mike_hearn 3y ago

The reason it's not discussed much is that what goes on downstream of tokenization is extremely opaque. It's lots of layers of the transformer network so the overall structure is documented but what exactly those numbers mean is hard to figure out. There's an article here where the structure of an image generation network is explored a bit: https://openai.com/research/sparse-transformer https://openai.com/research/sparse-transformer They have a visualization of what the different layers are paying attention to. There are also some good explanations of transformers elsewhere online. This one is old but I found it helpful: http://jalammar.github.io/illustrated-transformer/ http://jalammar.github.io/illustrated-transformer/

hsjqllzlfkf 3y ago
This was my suspicion, thank you.

SkyPuncher 3y ago

With all due respect, this feels like asking me to talk about math without talking about numbers. Tokens are so closely tied to modern LLMs that’s it’s basically impossible to not talk about them. They’re getting a lot of attention because they are the primitive. They’re the thing of most interest for improving performance.

wolfgang42 3y ago

You asked why there was a focus on discussing tokenizers, and got an answer explaining why tokenizers are something people would want to discuss.

washadjeffmad 3y ago
> ...It seems every other day there's a new article or implementation of a tokenizer on HN. But downstream from that, rarely anything. As a non-expert I would have thought to tokenizing is just one step. If someone points out a preponderance of information on one step relative to all other steps, they probably are not asking for even more information about that step. People like to chip in with what they've recently learned, so one answer is that most people on HN don't understand much beyond the input layer. A better answer is that the relative complexity of the processes in subsequent layers increases substantially, along with the requisite background to understand them. They also don't share the relative commonality of the input layer, so fewer people are qualified to discuss them with any authority. That's where I am, so I get it. I'm working on building learning resources for a symposium, and it feels very much like "Step 1: Tokenize, Step 2: ???, Step 3: Output!".