5 ms·

Tokenizers aren't considered the "sexy" part of LLMs, but where others see boring, I see opportunity. Papers like xVal[1], point toward specialization strategie

Tokenizers aren't considered the "sexy" part of LLMs, but where others see boring, I see opportunity. Papers like xVal[1], point toward specialization strategies in tokenization. Spelling and letter tasks are another problem that could benefit from innovation on the tokenization. LLMs are notoriously bad at counting letters in words or performing simply oulipos of letter omission. GPT-4o, for example, writes a small python program and executes it in order to count letter instances. We all know that tokenization effectively erases knowledge about letters in prompts and directly negatively impacts performance at these tasks, yet we haven't found a way to solve it. 1. https://ar5iv.labs.arxiv.org/html/2310.02989 https://ar5iv.labs.arxiv.org/html/2310.02989

IncreasePosts 2y ago
What's the issue with character-level tokenization(I assume this would be much better at count-the-letter tasks)? The article mentions it as an option but doesn't talk about why subword tokenization is preferred by most of the big LLMs out there.
- SEGyges 2y ago
  tokens are on average four characters and the number of residual streams (and therefore RAM) the LLM allocates to a given sequence is proportionate to the number of units of input. the flops is proportionate to their square in the attention calculation. you can hypothetically try to ameliorate this by other means, but if you just naively drop from tokenization to character or byte level models this is what goes wrong
  - p1esk 2y ago
    4x seq length expansion doesn’t sound that bad.
    - lechatonnoir 2y ago
      I mean, it's not completely fatal, but it means an approximately 16x increase in runtime cost, if I'm not mistaken. That's probably not worth trying to solve letter counting in most applications.
      - SEGyges 2y ago
        it is not necessarily 16x if you, e.g., decrease model width by a factor of 4 or so also, but yeah naively the RAM and FLOPs scale up by n^2
- Centigonal 2y ago
  I think it has to do with both performance (smaller tokens means more tokens per sentence read and more runs per sentence generated) and with how embeddings work. You need a token for "dog" and a token for "puppy" to represent the relationship between the two as a dimension in latent space.
- stephantul 2y ago
  Using subwords makes your sequences shorter, which makes them cost less. Besides that, for alphabetic languages, there exists almost no relation between form and meaning. I.e.: “ring” and “wing” differ by one letter but have no real common meaning. By picking the character or byte as your choice of representation, the model basically has to learn to distinguish ring and wing in context. This is a lot of work! So, while working on the character or byte level saves you some embeddings and thus makes your model smaller, it puts all of the work of distinguishing similar sequences with divergent meanings on the model itself, which means you need a larger model. By having subwords, a part of this distinguishing work already has been done by the vocabulary itself. As the article points out, this sometimes fails.
  - bunderbunder 2y ago
    I suspect that the holy grail here is figuring out how to break the input into a sequence of morphemes and non-morpheme lexical units.
    - thaumasiotes 2y ago
      What do you mean by non-morpheme lexical units? Syntactic particles, units too small to be morphemes? Lexical items that contain multiple morphemes? In either case, isn't this something we already do well?
      - bunderbunder 2y ago
        Punctuation, for example. And no, at least for the languages with which I'm familiar SOTA tokenizers tend to only capture the easy cases. For example, the GPT-4 tokenizer breaks the first sentence of your post like so: What/ do/ you/ mean/ by/ non/-m/orp/heme/ lexical/ units/? Notice how "morpheme" gets broken into three tokens, and none of them matches "morpheme"'s two morphemes. "Lexical" and "units" are each a single token, when they have three and two morphemes respectively. Or in French, the word "cafetière" gets chopped willy-nilly into "c/afet/ière". The canonical breakdown is "cafe/t/ière".
  - 2y ago
- cma 2y ago
  Context length performance and memory scales N^2. Smaller tokens mean worse scaling, up to a point.
kaycebasques 2y ago
> but where others see boring, I see opportunity I feel this way about embeddings This line of thought seems related to the old wisdom of finding innovative solutions by mucking around in the layer below whatever the "tools of the trade" are for your domain
screye 2y ago
Tokenizers face an odd compute issue. Since they're part of the pre-processing pipeline, you can't quickly test them out for effectiveness. You have to restart a pretraining run to test downstream effectiveness. Separately, As much as an attention module can do universal nonlinear transformations....I wonder if it makes sense to add specifuc modules for some math primitives as well. I remember that the executor paper [1] (slightly precursor to the attention is allyou need paper) created self contained modules for operations like less than, count, sum and then explicitly orchestrated them in the decoder. I'm surprised we haven't seen such solutions produce sota results from math-ai or code-ai research communities. [1] https://arxiv.org/abs/1705.03633 https://arxiv.org/abs/1705.03633
doctorpangloss 2y ago
> LLMs are notoriously bad at counting letters in words or performing simply oulipos of letter omission. If it were so simple, why hasn’t this already been dealt with? Multimodal VQA models also have had a hard time generalizing counting. Counting is not as simple as changing the tokenizer.
- kelseyfrog 2y ago
  I'm saying the oulipo rule is simple, not the task given current tokenization methods
- danielmarkbruce 2y ago
  Should the number 23 be tokenized as one token or two tokens?
  - tomrod 2y ago
    We already solved that with binary representation ;-)
  - thaumasiotes 2y ago
    Two. That's the reality. You interpret the token sequence by constructing a parse tree, but that doesn't require you to forget that the tokens exist.
    - danielmarkbruce 2y ago
      If you use standard BPE, you likely won't tokenize every number by it's digits, depending on the data set used to create the tokenizer. The point is, you have a choice. You can do the tokenization however you like. The reason 23 is interesting is that there is a case to be made that a model will more likely understand 23 is related to Jordan if it's one token, and if it's two tokens it's more difficult. The opposite is true for math problems. The reality is whatever we want to make it. It's likely that current schemes are... sub optimal. In practice it would be great if every token was geometrically well spaced after embedding, and preserve semantic information, among other things. The "other things" have taken precedent thus far.
Der_Einzige 2y ago
I wrote a whole paper about this exact topic! (Syntactic, phonetic, and related constraints) https://aclanthology.org/2022.cai-1.2/ https://aclanthology.org/2022.cai-1.2/
bunderbunder 2y ago
This was ages ago, in the pre-transformer era, and I can't find the link anymore. But once upon a time I read a great paper that demonstrated that most of the performance differences being reported among popular embedding models of the time were better explained by text cleaning and tokenization than they were by the embedding model itself. In other words, if you train a model using word2vec's preprocessing and GloVe's algorithm, the result looks more like a "standard-issue" word2vec model than a "standard-issue" GloVe model.
- authorfly 2y ago
  Yes those models were sensitive to the preprocessing, far more than transformers. However Word2Vec and GloVe were fundamentally different, when used as designed GloVe worked better pretty uniformly.
danielmarkbruce 2y ago
And decoders.
Suppafly 2y ago
>oulipos ?

IncreasePosts 2y ago

What's the issue with character-level tokenization(I assume this would be much better at count-the-letter tasks)? The article mentions it as an option but doesn't talk about why subword tokenization is preferred by most of the big LLMs out there.

SEGyges 2y ago
tokens are on average four characters and the number of residual streams (and therefore RAM) the LLM allocates to a given sequence is proportionate to the number of units of input. the flops is proportionate to their square in the attention calculation. you can hypothetically try to ameliorate this by other means, but if you just naively drop from tokenization to character or byte level models this is what goes wrong
- p1esk 2y ago
  4x seq length expansion doesn’t sound that bad.
  - lechatonnoir 2y ago
    I mean, it's not completely fatal, but it means an approximately 16x increase in runtime cost, if I'm not mistaken. That's probably not worth trying to solve letter counting in most applications.
    - SEGyges 2y ago
      it is not necessarily 16x if you, e.g., decrease model width by a factor of 4 or so also, but yeah naively the RAM and FLOPs scale up by n^2
Centigonal 2y ago
I think it has to do with both performance (smaller tokens means more tokens per sentence read and more runs per sentence generated) and with how embeddings work. You need a token for "dog" and a token for "puppy" to represent the relationship between the two as a dimension in latent space.
stephantul 2y ago
Using subwords makes your sequences shorter, which makes them cost less. Besides that, for alphabetic languages, there exists almost no relation between form and meaning. I.e.: “ring” and “wing” differ by one letter but have no real common meaning. By picking the character or byte as your choice of representation, the model basically has to learn to distinguish ring and wing in context. This is a lot of work! So, while working on the character or byte level saves you some embeddings and thus makes your model smaller, it puts all of the work of distinguishing similar sequences with divergent meanings on the model itself, which means you need a larger model. By having subwords, a part of this distinguishing work already has been done by the vocabulary itself. As the article points out, this sometimes fails.
- bunderbunder 2y ago
  I suspect that the holy grail here is figuring out how to break the input into a sequence of morphemes and non-morpheme lexical units.
  - thaumasiotes 2y ago
    What do you mean by non-morpheme lexical units? Syntactic particles, units too small to be morphemes? Lexical items that contain multiple morphemes? In either case, isn't this something we already do well?
    - bunderbunder 2y ago
      Punctuation, for example. And no, at least for the languages with which I'm familiar SOTA tokenizers tend to only capture the easy cases. For example, the GPT-4 tokenizer breaks the first sentence of your post like so: What/ do/ you/ mean/ by/ non/-m/orp/heme/ lexical/ units/? Notice how "morpheme" gets broken into three tokens, and none of them matches "morpheme"'s two morphemes. "Lexical" and "units" are each a single token, when they have three and two morphemes respectively. Or in French, the word "cafetière" gets chopped willy-nilly into "c/afet/ière". The canonical breakdown is "cafe/t/ière".
- 2y ago
cma 2y ago
Context length performance and memory scales N^2. Smaller tokens mean worse scaling, up to a point.

kaycebasques 2y ago

> but where others see boring, I see opportunity I feel this way about embeddings This line of thought seems related to the old wisdom of finding innovative solutions by mucking around in the layer below whatever the "tools of the trade" are for your domain

screye 2y ago

Tokenizers face an odd compute issue. Since they're part of the pre-processing pipeline, you can't quickly test them out for effectiveness. You have to restart a pretraining run to test downstream effectiveness. Separately, As much as an attention module can do universal nonlinear transformations....I wonder if it makes sense to add specifuc modules for some math primitives as well. I remember that the executor paper [1] (slightly precursor to the attention is allyou need paper) created self contained modules for operations like less than, count, sum and then explicitly orchestrated them in the decoder. I'm surprised we haven't seen such solutions produce sota results from math-ai or code-ai research communities. [1] https://arxiv.org/abs/1705.03633 https://arxiv.org/abs/1705.03633

doctorpangloss 2y ago

> LLMs are notoriously bad at counting letters in words or performing simply oulipos of letter omission. If it were so simple, why hasn’t this already been dealt with? Multimodal VQA models also have had a hard time generalizing counting. Counting is not as simple as changing the tokenizer.

kelseyfrog 2y ago
I'm saying the oulipo rule is simple, not the task given current tokenization methods
danielmarkbruce 2y ago
Should the number 23 be tokenized as one token or two tokens?
- tomrod 2y ago
  We already solved that with binary representation ;-)
- thaumasiotes 2y ago
  Two. That's the reality. You interpret the token sequence by constructing a parse tree, but that doesn't require you to forget that the tokens exist.
  - danielmarkbruce 2y ago
    If you use standard BPE, you likely won't tokenize every number by it's digits, depending on the data set used to create the tokenizer. The point is, you have a choice. You can do the tokenization however you like. The reason 23 is interesting is that there is a case to be made that a model will more likely understand 23 is related to Jordan if it's one token, and if it's two tokens it's more difficult. The opposite is true for math problems. The reality is whatever we want to make it. It's likely that current schemes are... sub optimal. In practice it would be great if every token was geometrically well spaced after embedding, and preserve semantic information, among other things. The "other things" have taken precedent thus far.

Der_Einzige 2y ago

I wrote a whole paper about this exact topic! (Syntactic, phonetic, and related constraints) https://aclanthology.org/2022.cai-1.2/ https://aclanthology.org/2022.cai-1.2/

bunderbunder 2y ago

This was ages ago, in the pre-transformer era, and I can't find the link anymore. But once upon a time I read a great paper that demonstrated that most of the performance differences being reported among popular embedding models of the time were better explained by text cleaning and tokenization than they were by the embedding model itself. In other words, if you train a model using word2vec's preprocessing and GloVe's algorithm, the result looks more like a "standard-issue" word2vec model than a "standard-issue" GloVe model.

authorfly 2y ago
Yes those models were sensitive to the preprocessing, far more than transformers. However Word2Vec and GloVe were fundamentally different, when used as designed GloVe worked better pretty uniformly.

danielmarkbruce 2y ago

And decoders.

Suppafly 2y ago

>oulipos ?