4 ms·

Huh, unless the demo is broken it seems to tokenize unicode per-byte, rather than per-character. E.g.: "あ" => [40948] "亜" => [12859, 250] "ｱ" => [

Huh, unless the demo is broken it seems to tokenize unicode per-byte, rather than per-character. E.g.: "あ" => [40948] "亜" => [12859, 250] "ｱ" => [171, 121, 109]

dudeinjapan 4y ago
Yeah I noticed similar. That can't be right...
- rhdunn 4y ago
  It's probably operating on UTF-8 data on a byte-per-byte level without any additional processing. Just feeding it the raw string data and letting it assign the tokens. It's similar to how it is splitting words at arbitrary points, rather than at clear morphological or lexical locations (e.g. on the Jane Austen text `"Now, ma'am," said Jane to her aunt, "shall we join Mrs. Elton?"` I've seen it tokenize that as `"|Now|,| ma|'|am|,"| said| Jane| to| her| aunt|,| "|shall| we| join| Mrs|.| El|ton|?"`).
  - dudeinjapan 4y ago
    I would find that hard to believe, as the bytes have zero semantic meaning, and moreover, pairing the wrong bytes in the output will result in complete gibberish. It would be akin to tokenizing each English letter "N|o|w|,| |m|a|'|a|m|..." except far worse. Moreover it's trivially easy to tokenize the glyphs.
    - pyentropy 4y ago
      A character is the base unit of written communication. Single characters as tokens is not a bad idea, it just requires too much resources to make it learn and infer. BPE is a tradeoff between single letters (computationally hard) and a word dictionary (can't handle novel words, languages or complex structures like code syntax). Note that tokens must be hardcoded because the neural network has an output layer consisting of neurons one-to-one mapped to the tokens (and the predicted word is the most activated neuron). Human brains roughly do the same thing - that's why we have syllables as a tradeoff between letters and words.
      - dudeinjapan 4y ago
        > A character is the base unit of written communication Yes, I guess the point here is that the glyph, not the byte, is the base unit of communication in Unicode charsets.
numpad0 4y ago
"０１２３４５６７８９" => [ 171, 120, 238, 171, 120, 239, 171, 120, 240, 171, 120, 241, 171, 120, 242, 171, 120, 243, 171, 120, 244, 171, 120, 245, 171, 120, 246, 171, 120, 247 ] "～" => [171, 121, 252, 198] Whoa. Literally just giving it "Ｐｏｔａｔｏ" gives thrice as much token count as the letter count, 18 tokens for 6 letters.

Huh, unless the demo is broken it seems to tokenize unicode per-byte, rather than per-character. E.g.: "あ" => [40948] "亜" => [12859, 250] "ｱ" => [171, 121, 109]

dudeinjapan 4y ago

Yeah I noticed similar. That can't be right...

rhdunn 4y ago
It's probably operating on UTF-8 data on a byte-per-byte level without any additional processing. Just feeding it the raw string data and letting it assign the tokens. It's similar to how it is splitting words at arbitrary points, rather than at clear morphological or lexical locations (e.g. on the Jane Austen text `"Now, ma'am," said Jane to her aunt, "shall we join Mrs. Elton?"` I've seen it tokenize that as `"|Now|,| ma|'|am|,"| said| Jane| to| her| aunt|,| "|shall| we| join| Mrs|.| El|ton|?"`).
- dudeinjapan 4y ago
  I would find that hard to believe, as the bytes have zero semantic meaning, and moreover, pairing the wrong bytes in the output will result in complete gibberish. It would be akin to tokenizing each English letter "N|o|w|,| |m|a|'|a|m|..." except far worse. Moreover it's trivially easy to tokenize the glyphs.
  - pyentropy 4y ago
    A character is the base unit of written communication. Single characters as tokens is not a bad idea, it just requires too much resources to make it learn and infer. BPE is a tradeoff between single letters (computationally hard) and a word dictionary (can't handle novel words, languages or complex structures like code syntax). Note that tokens must be hardcoded because the neural network has an output layer consisting of neurons one-to-one mapped to the tokens (and the predicted word is the most activated neuron). Human brains roughly do the same thing - that's why we have syllables as a tradeoff between letters and words.
    - dudeinjapan 4y ago
      > A character is the base unit of written communication Yes, I guess the point here is that the glyph, not the byte, is the base unit of communication in Unicode charsets.

numpad0 4y ago

"０１２３４５６７８９" => [ 171, 120, 238, 171, 120, 239, 171, 120, 240, 171, 120, 241, 171, 120, 242, 171, 120, 243, 171, 120, 244, 171, 120, 245, 171, 120, 246, 171, 120, 247 ] "～" => [171, 121, 252, 198] Whoa. Literally just giving it "Ｐｏｔａｔｏ" gives thrice as much token count as the letter count, 18 tokens for 6 letters.