3 ms·
I would find that hard to believe, as the bytes have zero semantic meaning, and moreover, pairing the wrong bytes in the output will result in complete gibberis
by dudeinjapan 4y ago
I would find that hard to believe, as the bytes have zero semantic meaning, and moreover, pairing the wrong bytes in the output will result in complete gibberish. It would be akin to tokenizing each English letter "N|o|w|,| |m|a|'|a|m|..." except far worse.
Moreover it's trivially easy to tokenize the glyphs.
- pyentropy 4y agoA character is the base unit of written communication. Single characters as tokens is not a bad idea, it just requires too much resources to make it learn and infer. BPE is a tradeoff between single letters (computationally hard) and a word dictionary (can't handle novel words, languages or complex structures like code syntax). Note that tokens must be hardcoded because the neural network has an output layer consisting of neurons one-to-one mapped to the tokens (and the predicted word is the most activated neuron). Human brains roughly do the same thing - that's why we have syllables as a tradeoff between letters and words.
- dudeinjapan 4y ago> A character is the base unit of written communication Yes, I guess the point here is that the glyph, not the byte, is the base unit of communication in Unicode charsets.