4 ms·

Tokens are the primitives that most LLMs (and broadly a lot of NLP) works with. While, you and I would expect whole-words to be tokens, many tokens are shorter

Tokens are the primitives that most LLMs (and broadly a lot of NLP) works with. While, you and I would expect whole-words to be tokens, many tokens are shorter - 3 to 4 characters - and don't always match the sentence structure you and I expect. This can create some interesting challenges and unexpected behavior. It also makes certain things, like vectorization, a challenge since tokens may not map 1:1 with the words you intend to weight them against.

whimsicalism 3y ago
You are using the word vectorization in an idiosyncratic way, you are referring to the process of embedding words?
- SkyPuncher 3y ago
  Because I’m not an expert in this area. I know it well enough to build products around it, but it’s not my deep area of expertise. Just trying to provide an example.
hsjqllzlfkf 3y ago
Your answer explains what tokenizers are, which isn't what I asked. You also told me something interesting about tokenizers, which is also not what I asked. Can you tell me anything NOT about tokenized? This is my point.
- mike_hearn 3y ago
  The reason it's not discussed much is that what goes on downstream of tokenization is extremely opaque. It's lots of layers of the transformer network so the overall structure is documented but what exactly those numbers mean is hard to figure out. There's an article here where the structure of an image generation network is explored a bit: https://openai.com/research/sparse-transformer https://openai.com/research/sparse-transformer They have a visualization of what the different layers are paying attention to. There are also some good explanations of transformers elsewhere online. This one is old but I found it helpful: http://jalammar.github.io/illustrated-transformer/ http://jalammar.github.io/illustrated-transformer/
  - hsjqllzlfkf 3y ago
    This was my suspicion, thank you.
- SkyPuncher 3y ago
  With all due respect, this feels like asking me to talk about math without talking about numbers. Tokens are so closely tied to modern LLMs that’s it’s basically impossible to not talk about them. They’re getting a lot of attention because they are the primitive. They’re the thing of most interest for improving performance.
- wolfgang42 3y ago
  You asked why there was a focus on discussing tokenizers, and got an answer explaining why tokenizers are something people would want to discuss.
  - washadjeffmad 3y ago
    > ...It seems every other day there's a new article or implementation of a tokenizer on HN. But downstream from that, rarely anything. As a non-expert I would have thought to tokenizing is just one step. If someone points out a preponderance of information on one step relative to all other steps, they probably are not asking for even more information about that step. People like to chip in with what they've recently learned, so one answer is that most people on HN don't understand much beyond the input layer. A better answer is that the relative complexity of the processes in subsequent layers increases substantially, along with the requisite background to understand them. They also don't share the relative commonality of the input layer, so fewer people are qualified to discuss them with any authority. That's where I am, so I get it. I'm working on building learning resources for a symposium, and it feels very much like "Step 1: Tokenize, Step 2: ???, Step 3: Output!".
thaumasiotes 3y ago
> While, you and I would expect whole-words to be tokens, many tokens are shorter - 3 to 4 characters - and don't always match the sentence structure you and I expect. There is a phenomenon called Broca's Aphasia which is, essentially, the inability to connect words into sentences. This mostly prevents the patient from communicating via language. But patients with this condition can reveal quite a bit about the structure of the language they can no longer speak. One example discussed in The Language Instinct is someone who works at (and was injured at) a mill. He is unable to produce utterances that are more than one word long, though he seems to do well at understanding what people say to him. One of his single-word utterances, describing the mill where he works, is "Four hundred tons a day!". This is the opposite of what you describe, a single token that is longer than one word in the base language instead of being shorter. But it appears to be the same kind of thing. By the way, if you study a highly inflectional language such as Latin or Russian, you will lose the assumption that interpretive tokens should be whole words. You'd still expect them to align closely with sentence structure, though.
- doormatt 3y ago
  >One of his single-word utterances, describing the mill where he works, is "Four hundred tons a day!". I'm sorry, but I'm lost on how that's a single-word utterance.
- wolfgang42 3y ago
  You can observe (what I assume is) the same tokenization phenomenon in people who are struggling to speak (for example because they’re distracted by something or not native speakers): stock fragments will come out all at once, and less common words will get split, usually on affixes or at the join point of compound words.

whimsicalism 3y ago

You are using the word vectorization in an idiosyncratic way, you are referring to the process of embedding words?

SkyPuncher 3y ago
Because I’m not an expert in this area. I know it well enough to build products around it, but it’s not my deep area of expertise. Just trying to provide an example.

hsjqllzlfkf 3y ago

Your answer explains what tokenizers are, which isn't what I asked. You also told me something interesting about tokenizers, which is also not what I asked. Can you tell me anything NOT about tokenized? This is my point.

mike_hearn 3y ago
The reason it's not discussed much is that what goes on downstream of tokenization is extremely opaque. It's lots of layers of the transformer network so the overall structure is documented but what exactly those numbers mean is hard to figure out. There's an article here where the structure of an image generation network is explored a bit: https://openai.com/research/sparse-transformer https://openai.com/research/sparse-transformer They have a visualization of what the different layers are paying attention to. There are also some good explanations of transformers elsewhere online. This one is old but I found it helpful: http://jalammar.github.io/illustrated-transformer/ http://jalammar.github.io/illustrated-transformer/
- hsjqllzlfkf 3y ago
  This was my suspicion, thank you.
SkyPuncher 3y ago
With all due respect, this feels like asking me to talk about math without talking about numbers. Tokens are so closely tied to modern LLMs that’s it’s basically impossible to not talk about them. They’re getting a lot of attention because they are the primitive. They’re the thing of most interest for improving performance.
wolfgang42 3y ago
You asked why there was a focus on discussing tokenizers, and got an answer explaining why tokenizers are something people would want to discuss.
- washadjeffmad 3y ago
  > ...It seems every other day there's a new article or implementation of a tokenizer on HN. But downstream from that, rarely anything. As a non-expert I would have thought to tokenizing is just one step. If someone points out a preponderance of information on one step relative to all other steps, they probably are not asking for even more information about that step. People like to chip in with what they've recently learned, so one answer is that most people on HN don't understand much beyond the input layer. A better answer is that the relative complexity of the processes in subsequent layers increases substantially, along with the requisite background to understand them. They also don't share the relative commonality of the input layer, so fewer people are qualified to discuss them with any authority. That's where I am, so I get it. I'm working on building learning resources for a symposium, and it feels very much like "Step 1: Tokenize, Step 2: ???, Step 3: Output!".

thaumasiotes 3y ago

> While, you and I would expect whole-words to be tokens, many tokens are shorter - 3 to 4 characters - and don't always match the sentence structure you and I expect. There is a phenomenon called Broca's Aphasia which is, essentially, the inability to connect words into sentences. This mostly prevents the patient from communicating via language. But patients with this condition can reveal quite a bit about the structure of the language they can no longer speak. One example discussed in The Language Instinct is someone who works at (and was injured at) a mill. He is unable to produce utterances that are more than one word long, though he seems to do well at understanding what people say to him. One of his single-word utterances, describing the mill where he works, is "Four hundred tons a day!". This is the opposite of what you describe, a single token that is longer than one word in the base language instead of being shorter. But it appears to be the same kind of thing. By the way, if you study a highly inflectional language such as Latin or Russian, you will lose the assumption that interpretive tokens should be whole words. You'd still expect them to align closely with sentence structure, though.

doormatt 3y ago
>One of his single-word utterances, describing the mill where he works, is "Four hundred tons a day!". I'm sorry, but I'm lost on how that's a single-word utterance.
wolfgang42 3y ago
You can observe (what I assume is) the same tokenization phenomenon in people who are struggling to speak (for example because they’re distracted by something or not native speakers): stock fragments will come out all at once, and less common words will get split, usually on affixes or at the join point of compound words.