3 ms·

From the OP, it sounds like those tokens are generated from the UTF-8 bytes instead of from the Unicode code points. And those bytes are, taken in isolation, co

by amluto 2y ago

From the OP, it sounds like those tokens are generated from the UTF-8 bytes instead of from the Unicode code points. And those bytes are, taken in isolation, complete nonsense. Imagine a token that represented the right side of the letter d followed by the left side of the letter e but could also represent other mishmashes of characters. I bet the first layer of the model is mostly stuck reconstructing something resembling actual words. (UTF-8 is locally decidable. I bet that a bit of work on the token list could cause it to avoid tokens that do not align with code point boundaries.)

pama 2y ago
To be clear these tokenizers use byte-pair encoding (subword tokens) so an individual token index typically corresponds to a piece of a word; this index does not depend on any intermediate decoding of the byte stream as long as the start of the stream is a start of your input. The decoding always works left to right and always starts at the start of the stream. You could write a tokenizer that uses plain bytes and one that uses unicode code points if your tokenizer was trained on unicode and forced to keep unicode codes together (almost all are), and the results would be identical for all practical purposes.
- mmoskal 2y ago
  llama2, llama3, gpt3, and gpt4 tokenizers are all trained on UTF8 bytes and all include invalid (partial) UTF8 tokens. For llama3 it's only 256 tokens, one for each byte, but for the others it's more interesting (eg., 1361 tokens with UTF8 fragments in llama3). There is over 1M possible Unicode code points, and 150k actually defined. Thus, you can't really encode all of them with splitting.
  - pama 2y ago
    Right. I stand corrected and it makes sense. I may have misunderstood the OP. It would not make sense to encode each unicode point as a token and then start training the subword tokenizer on top unless we go to the vocabularies of many millions.
  - yorwba 2y ago
    There might be better ways to split than simply using bytes, though. Normalizing Unicode to NFD form and replacing CJK characters with their Ideographic Description Sequences gets me down to slightly more than 50k codepoints, and visual inspection indicates that the IDS library I used is missing data for quite a few characters, so maybe 40k or so is possible. Then you could have the "how many r in strawberry" equivalent of "how many 月 in 明月清风"! On the negative side, a model trained on such a representation could make up CJK characters not in Unicode and you would need a procedural font to display them properly.
mmoskal 2y ago
You essentially have to run a byte regular expression that enforces valid UTF8. When you take into account exclusion for surrogate pairs and overlongs, you end up with about 14 states in the corresponding automaton. This is one thing among many done by our llguidance [0] library. [0] https://github.com/microsoft/llguidance https://github.com/microsoft/llguidance edit: if anyone's interested: (([C2-DF] [80-BF]) | (E0 [A0-BF] [80-BF]) | ([E1-EC] [80-BF] [80-BF]) | (ED [80-9F] [80-BF]) | ([EE-EF] [80-BF] [80-BF]) | (F0 [90-BF] [80-BF] [80-BF]) | ([F1-F3] [80-BF] [80-BF] [80-BF]) | (F4 [80-8F] [80-BF] [80-BF]) | [00-7F])
- pama 2y ago
  Cool repo—thanks!

by amluto 2y ago

pama 2y ago

To be clear these tokenizers use byte-pair encoding (subword tokens) so an individual token index typically corresponds to a piece of a word; this index does not depend on any intermediate decoding of the byte stream as long as the start of the stream is a start of your input. The decoding always works left to right and always starts at the start of the stream. You could write a tokenizer that uses plain bytes and one that uses unicode code points if your tokenizer was trained on unicode and forced to keep unicode codes together (almost all are), and the results would be identical for all practical purposes.

mmoskal 2y ago
llama2, llama3, gpt3, and gpt4 tokenizers are all trained on UTF8 bytes and all include invalid (partial) UTF8 tokens. For llama3 it's only 256 tokens, one for each byte, but for the others it's more interesting (eg., 1361 tokens with UTF8 fragments in llama3). There is over 1M possible Unicode code points, and 150k actually defined. Thus, you can't really encode all of them with splitting.
- pama 2y ago
  Right. I stand corrected and it makes sense. I may have misunderstood the OP. It would not make sense to encode each unicode point as a token and then start training the subword tokenizer on top unless we go to the vocabularies of many millions.
- yorwba 2y ago
  There might be better ways to split than simply using bytes, though. Normalizing Unicode to NFD form and replacing CJK characters with their Ideographic Description Sequences gets me down to slightly more than 50k codepoints, and visual inspection indicates that the IDS library I used is missing data for quite a few characters, so maybe 40k or so is possible. Then you could have the "how many r in strawberry" equivalent of "how many 月 in 明月清风"! On the negative side, a model trained on such a representation could make up CJK characters not in Unicode and you would need a procedural font to display them properly.

mmoskal 2y ago

You essentially have to run a byte regular expression that enforces valid UTF8. When you take into account exclusion for surrogate pairs and overlongs, you end up with about 14 states in the corresponding automaton. This is one thing among many done by our llguidance [0] library. [0] https://github.com/microsoft/llguidance https://github.com/microsoft/llguidance edit: if anyone's interested: (([C2-DF] [80-BF]) | (E0 [A0-BF] [80-BF]) | ([E1-EC] [80-BF] [80-BF]) | (ED [80-9F] [80-BF]) | ([EE-EF] [80-BF] [80-BF]) | (F0 [90-BF] [80-BF] [80-BF]) | ([F1-F3] [80-BF] [80-BF] [80-BF]) | (F4 [80-8F] [80-BF] [80-BF]) | [00-7F])

pama 2y ago
Cool repo—thanks!