4 ms·

> The most obvious thing differentiating DeepSeek’s tokenizer from other’s is a substantial fraction of the training data being Chinese. This makes things much

by amluto 2y ago

> The most obvious thing differentiating DeepSeek’s tokenizer from other’s is a substantial fraction of the training data being Chinese. This makes things much more difficult to work with — tokenizations are learned at the byte level, but UTF-8 Chinese characters are usually several bytes long. I realize that these models are more than powerful enough to deal with this nonsense, but it seems like, especially for smaller models, it might make sense to try using the Unicode input as such instead of treating it as bytes.

pama 2y ago
Not sure what you mean here—care to elaborate? The eventual input to these models are integer token IDs (128k different ones for DeepSeek). The tokenizers do the conversions from Unicode streams to streams of token IDs.
- cchance 2y ago
  I still wonder why we're training models on all these languages especially when they have different alphabets etc, we've got solid translators, wouldn't it be more parameter dense to target one language for all data and tokens, and then have a layer specifically for input and output translation?
  - eightysixfour 2y ago
    I'd be interested to know if adding more languages makes them more or less performant. It is my understanding that you have to add code for the models to perform well, for example.
    - cchance 2y ago
      My issue and questions isn't so much languages but character sets, expanding say an ascii dataset to include all the non-ascii language glyphs or other unicode characters has got to balloon the parameter count
  - ijustlovemath 2y ago
    more languages gives deeper semantic understanding; I think it only helps with diversity of data, which ultimately improves outputs
  - sva_ 2y ago
    You'll have a lower bound on the quality of the translator you're using. There's an idea that you can generalize concepts among different languages, and that you'll benefit from the extended training corpus. As in, talking about an idea from different perspectives helps the model carve it out. But I don't have anything concrete to back that claim up.
    - econ 2y ago
      Each language has unique tools. If you have a word for something or even better a whole set of words the conversation works a lot better than in a language that has nothing of the kind. English elaborately talks about all kinds of Communities. Dutch does have a word like it but it is almost never used. Or, how should we talk about the kind of snow if we have only one word? https://watchingtheswedes.com/2018/02/28/50-words-for-snow/ https://watchingtheswedes.com/2018/02/28/50-words-for-snow/
  - cma 2y ago
    More languages helps it at novel translation tasks, models have been tested with languages not in/barely in the corpus and a translation book in context and were able to do an ok job. You'll also have things like mulimodal where you want to preserve all the tonality and emphasis in the input language.
  - astrange 2y ago
    Bitter lesson says any kind of specialization is not worth it[0]. Also, you want to be able to have mixed language conversations, like defining a Chinese word in English. [0] but it might be worth it if you need a smaller model because then there are tradeoffs again.
- amluto 2y ago
  From the OP, it sounds like those tokens are generated from the UTF-8 bytes instead of from the Unicode code points. And those bytes are, taken in isolation, complete nonsense. Imagine a token that represented the right side of the letter d followed by the left side of the letter e but could also represent other mishmashes of characters. I bet the first layer of the model is mostly stuck reconstructing something resembling actual words. (UTF-8 is locally decidable. I bet that a bit of work on the token list could cause it to avoid tokens that do not align with code point boundaries.)
  - pama 2y ago
    To be clear these tokenizers use byte-pair encoding (subword tokens) so an individual token index typically corresponds to a piece of a word; this index does not depend on any intermediate decoding of the byte stream as long as the start of the stream is a start of your input. The decoding always works left to right and always starts at the start of the stream. You could write a tokenizer that uses plain bytes and one that uses unicode code points if your tokenizer was trained on unicode and forced to keep unicode codes together (almost all are), and the results would be identical for all practical purposes.
    - mmoskal 2y ago
      llama2, llama3, gpt3, and gpt4 tokenizers are all trained on UTF8 bytes and all include invalid (partial) UTF8 tokens. For llama3 it's only 256 tokens, one for each byte, but for the others it's more interesting (eg., 1361 tokens with UTF8 fragments in llama3). There is over 1M possible Unicode code points, and 150k actually defined. Thus, you can't really encode all of them with splitting.
      - pama 2y ago
        Right. I stand corrected and it makes sense. I may have misunderstood the OP. It would not make sense to encode each unicode point as a token and then start training the subword tokenizer on top unless we go to the vocabularies of many millions.
      - 2y ago
- singularity2001 2y ago
  probably something as byte latent tokenization, or get rid of organization altogether as kaparthy suggested
brookst 2y ago
“Tokenizations are learned at the byte level” seems wrong. Tokens are integer representations of one or more characters, which themselves can be multiple bytes. When you tokenize “ant” to 0x38 0xF9, it doesn’t matter if the original was three bytes of ascii or 0x00 0x00 0x00 0x61 0x00 0x00 0x00 0x6E 0x00 0x00 0x00 0x74
- mmoskal 2y ago
  Tokens are in fact sequences of bytes not characters. For example, llama3 tokenizer (128k tokens) includes 1361 tokens that are invalid UTF8 (or rather they are partial UTF8). Models will generally only produce valid UTF8 (that is when bytes of tokens are concatenated they are valid UTF8), unless really confused.
  - petters 2y ago
    They are, but they should not be
bhuztez 2y ago
Being a fanboy of Universal(统一) Token(文字)， I think Chinese is the most easy one to work with. Since Chinese has no characters, it just have a few thousand tokens. Unicode code point is good starting point for Chinese. What about English? Just as there is no natural boundary between tokens in English, there is no natural boundary between words in Chinese. Before LLM became popular, people had invented many ways to do Chinese word segmentation, just like nowadays people are inventing many ways to do tokenization. However in the past, most of the time, you would end up with ngrams. If we learn that from history, ngrams should be a good starting point for English. For example, word "token" should be 3 tokens, "tok", "oke", "ken". Once add Chinese, everything should be just fine. To be more controversial, I would say there is no such a language called Chinese. They are a group of languages who adopted Universal Token. Now it is time for English to jump on the bandwagon.
petters 2y ago
I completely agree! This is an oversight that should be fixed

by amluto 2y ago

pama 2y ago

Not sure what you mean here—care to elaborate? The eventual input to these models are integer token IDs (128k different ones for DeepSeek). The tokenizers do the conversions from Unicode streams to streams of token IDs.

cchance 2y ago
I still wonder why we're training models on all these languages especially when they have different alphabets etc, we've got solid translators, wouldn't it be more parameter dense to target one language for all data and tokens, and then have a layer specifically for input and output translation?
- eightysixfour 2y ago
  I'd be interested to know if adding more languages makes them more or less performant. It is my understanding that you have to add code for the models to perform well, for example.
  - cchance 2y ago
    My issue and questions isn't so much languages but character sets, expanding say an ascii dataset to include all the non-ascii language glyphs or other unicode characters has got to balloon the parameter count
- ijustlovemath 2y ago
  more languages gives deeper semantic understanding; I think it only helps with diversity of data, which ultimately improves outputs
- sva_ 2y ago
  You'll have a lower bound on the quality of the translator you're using. There's an idea that you can generalize concepts among different languages, and that you'll benefit from the extended training corpus. As in, talking about an idea from different perspectives helps the model carve it out. But I don't have anything concrete to back that claim up.
  - econ 2y ago
    Each language has unique tools. If you have a word for something or even better a whole set of words the conversation works a lot better than in a language that has nothing of the kind. English elaborately talks about all kinds of Communities. Dutch does have a word like it but it is almost never used. Or, how should we talk about the kind of snow if we have only one word? https://watchingtheswedes.com/2018/02/28/50-words-for-snow/ https://watchingtheswedes.com/2018/02/28/50-words-for-snow/
- cma 2y ago
  More languages helps it at novel translation tasks, models have been tested with languages not in/barely in the corpus and a translation book in context and were able to do an ok job. You'll also have things like mulimodal where you want to preserve all the tonality and emphasis in the input language.
- astrange 2y ago
  Bitter lesson says any kind of specialization is not worth it[0]. Also, you want to be able to have mixed language conversations, like defining a Chinese word in English. [0] but it might be worth it if you need a smaller model because then there are tradeoffs again.
amluto 2y ago
From the OP, it sounds like those tokens are generated from the UTF-8 bytes instead of from the Unicode code points. And those bytes are, taken in isolation, complete nonsense. Imagine a token that represented the right side of the letter d followed by the left side of the letter e but could also represent other mishmashes of characters. I bet the first layer of the model is mostly stuck reconstructing something resembling actual words. (UTF-8 is locally decidable. I bet that a bit of work on the token list could cause it to avoid tokens that do not align with code point boundaries.)
- pama 2y ago
  To be clear these tokenizers use byte-pair encoding (subword tokens) so an individual token index typically corresponds to a piece of a word; this index does not depend on any intermediate decoding of the byte stream as long as the start of the stream is a start of your input. The decoding always works left to right and always starts at the start of the stream. You could write a tokenizer that uses plain bytes and one that uses unicode code points if your tokenizer was trained on unicode and forced to keep unicode codes together (almost all are), and the results would be identical for all practical purposes.
  - mmoskal 2y ago
    llama2, llama3, gpt3, and gpt4 tokenizers are all trained on UTF8 bytes and all include invalid (partial) UTF8 tokens. For llama3 it's only 256 tokens, one for each byte, but for the others it's more interesting (eg., 1361 tokens with UTF8 fragments in llama3). There is over 1M possible Unicode code points, and 150k actually defined. Thus, you can't really encode all of them with splitting.
    - pama 2y ago
      Right. I stand corrected and it makes sense. I may have misunderstood the OP. It would not make sense to encode each unicode point as a token and then start training the subword tokenizer on top unless we go to the vocabularies of many millions.
    - 2y ago
singularity2001 2y ago
probably something as byte latent tokenization, or get rid of organization altogether as kaparthy suggested

brookst 2y ago

“Tokenizations are learned at the byte level” seems wrong. Tokens are integer representations of one or more characters, which themselves can be multiple bytes. When you tokenize “ant” to 0x38 0xF9, it doesn’t matter if the original was three bytes of ascii or 0x00 0x00 0x00 0x61 0x00 0x00 0x00 0x6E 0x00 0x00 0x00 0x74

mmoskal 2y ago
Tokens are in fact sequences of bytes not characters. For example, llama3 tokenizer (128k tokens) includes 1361 tokens that are invalid UTF8 (or rather they are partial UTF8). Models will generally only produce valid UTF8 (that is when bytes of tokens are concatenated they are valid UTF8), unless really confused.
- petters 2y ago
  They are, but they should not be

bhuztez 2y ago

Being a fanboy of Universal(统一) Token(文字)， I think Chinese is the most easy one to work with. Since Chinese has no characters, it just have a few thousand tokens. Unicode code point is good starting point for Chinese. What about English? Just as there is no natural boundary between tokens in English, there is no natural boundary between words in Chinese. Before LLM became popular, people had invented many ways to do Chinese word segmentation, just like nowadays people are inventing many ways to do tokenization. However in the past, most of the time, you would end up with ngrams. If we learn that from history, ngrams should be a good starting point for English. For example, word "token" should be 3 tokens, "tok", "oke", "ken". Once add Chinese, everything should be just fine. To be more controversial, I would say there is no such a language called Chinese. They are a group of languages who adopted Universal Token. Now it is time for English to jump on the bandwagon.

petters 2y ago

I completely agree! This is an oversight that should be fixed