3 ms·
“Tokenizations are learned at the byte level” seems wrong. Tokens are integer representations of one or more characters, which themselves can be multiple bytes.
by brookst 2y ago
“Tokenizations are learned at the byte level” seems wrong. Tokens are integer representations of one or more characters, which themselves can be multiple bytes.
When you tokenize “ant” to 0x38 0xF9, it doesn’t matter if the original was three bytes of ascii or 0x00 0x00 0x00 0x61 0x00 0x00 0x00 0x6E 0x00 0x00 0x00 0x74
- mmoskal 2y agoTokens are in fact sequences of bytes not characters. For example, llama3 tokenizer (128k tokens) includes 1361 tokens that are invalid UTF8 (or rather they are partial UTF8). Models will generally only produce valid UTF8 (that is when bytes of tokens are concatenated they are valid UTF8), unless really confused.
- petters 2y agoThey are, but they should not be