3 ms·

I'm reminded of the "Unicode Tags" craze from a few months ago. [1] It was discovered that some LLMs effortlessly understand characters from the "Tag" range in

by xg15 2y ago

I'm reminded of the "Unicode Tags" craze from a few months ago. [1] It was discovered that some LLMs effortlessly understand characters from the "Tag" range in Unicode and treat them like ASCII, even though those characters are used virtually nowhere in normal text and you in fact need specialized tools to just make them visible. (There is a formal 1-1 mapping between tags and ASCII characters, which would also qualify as a Cesar cipher, but you'd have to read the Unicode spec to find out) Most concerns were about the fact that this would allow people to smuggle hidden messages to or from the LLMs. But an interesting question was also how the models had even learned the mapping in the first place if tags never occurred in the training data anywhere. As I understood it, the solution was pretty simple though: They hadn't. There was no specialized circuit for tags in the models. Tag characters just had the property that if you write them as bytes, they will look like "<some prefix bytes> <byte pattern of the corresponding ASCII character>". So already the tokenizer would parse the characters as ASCII, interleaved with "unknown" tokens for the prefixes. All the model had to do was to ignore the "unknown" tokens and it could process the rest like ASCII. No Cesar cipher decoding needed! Are we sure something similar isn't happening here? [1] https://arstechnica.com/security/2024/10/ai-chatbots-can-read-and-write-invisible-text-creating-an-ideal-covert-channel/ https://arstechnica.com/security/2024/10/ai-chatbots-can-rea...

yorwba 2y ago
This is exactly what's happening here. But note that UTF-8 is self-synchronizing, so no encoding of one character contains the encoding of another as a subsequence. Instead, both tag characters and the Byzantine music notation in the article look like "<some prefix bytes> <byte pattern of the corresponding ASCII character + 96>" They share this property with the Fullwidth Latin block, which does occur in the wild interspersed with Japanese or Chinese text.
- xg15 2y ago
  Ah, that makes sense. Thank you!
- thaumasiotes 2y ago
  > They share this property with the Fullwidth Latin block, which does occur in the wild interspersed with Japanese or Chinese text. How common is that? In my experience it's much more normal for Chinese text to intersperse ordinary ascii characters. https://www.zdic.net/hans/%E8%84%B8 https://www.zdic.net/hans/%E8%84%B8 I'm not sure what purpose fullwidth characters are supposed to serve, but whatever it is, it doesn't seem like they're succeeding.
  - yorwba 2y ago
    Fullwidth Latin characters exist so that you can arrange your text into a grid without the occasional word in Latin script messing up your alignment. Most people don't really care about this, or, if they do, simply use a font that renders regular Latin at full width (or half width to be more space-efficient) but very occasionally the Fullwidth Latin codepoints get some use. It's more common in Japanese (though still rare) than Chinese in my experience, but e.g. the Project Gutenberg ebook of 阿Ｑ正傳 https://gutenberg.org/cache/epub/25332/pg25332-images.html https://gutenberg.org/cache/epub/25332/pg25332-images.html uses fullwidth Qs.
RedNifre 2y ago
This sounds odd, why would you feed the LLM text as bytes instead of characters?
- yorwba 2y ago
  Because if you start with characters, much of the token vocabulary would be dedicated to rare Chinese characters right off the bat. If you start from UTF-8 bytes, you can dedicate more token space to common sequences of multiple characters (i.e. words people actually use) and achieve much better compression ratios.
  - RedNifre 2y ago
    I don't understand. Why would much of the vocabulary be dedicated to rare Chinese characters? Wouldn't those need to show up in the training data first? And if they did, wouldn't they also show up as weird byte sequences? And aren't UTF-8 byte sequences kinda risky for everything other than ASCII, since only ASCII bytes and header bytes are unambiguous, whereas following bytes (10***) are very ambiguous individually? I mean, sure, the LLM would notice that their meaning changes depending on preceding following- and header-bytes, but it is still not clear to me, why UTF-8 bytes are better for LLMs than characters (or even grapheme clusters). UTF-8 bytes seem like a very arbitrary choice to me. Why not do UTF-9 instead and get the most important Latin letters as single ninebitbytes?
    - yorwba 2y ago
      Yes, rare Chinese characters do show up in the training data (the rarest of them at least appear in lists of characters) and yes, they get tokenized as weird byte sequences, making the model work harder to process them, but it's better for that to happen to rare characters than to common words. It's a tradeoff. And of course UTF-8 is unlikely to be the single best encoding (e.g. Anthropic has a tokenizer that turns all caps text into a special caps lock symbol plus the regular-case equivalent) but much of it is papered over by byte-pair encoding. E.g. the most important Latin letters appear often enough that they get dedicated tokens anyways.

by xg15 2y ago

yorwba 2y ago

This is exactly what's happening here. But note that UTF-8 is self-synchronizing, so no encoding of one character contains the encoding of another as a subsequence. Instead, both tag characters and the Byzantine music notation in the article look like "<some prefix bytes> <byte pattern of the corresponding ASCII character + 96>" They share this property with the Fullwidth Latin block, which does occur in the wild interspersed with Japanese or Chinese text.

xg15 2y ago
Ah, that makes sense. Thank you!
thaumasiotes 2y ago
> They share this property with the Fullwidth Latin block, which does occur in the wild interspersed with Japanese or Chinese text. How common is that? In my experience it's much more normal for Chinese text to intersperse ordinary ascii characters. https://www.zdic.net/hans/%E8%84%B8 https://www.zdic.net/hans/%E8%84%B8 I'm not sure what purpose fullwidth characters are supposed to serve, but whatever it is, it doesn't seem like they're succeeding.
- yorwba 2y ago
  Fullwidth Latin characters exist so that you can arrange your text into a grid without the occasional word in Latin script messing up your alignment. Most people don't really care about this, or, if they do, simply use a font that renders regular Latin at full width (or half width to be more space-efficient) but very occasionally the Fullwidth Latin codepoints get some use. It's more common in Japanese (though still rare) than Chinese in my experience, but e.g. the Project Gutenberg ebook of 阿Ｑ正傳 https://gutenberg.org/cache/epub/25332/pg25332-images.html https://gutenberg.org/cache/epub/25332/pg25332-images.html uses fullwidth Qs.

RedNifre 2y ago

This sounds odd, why would you feed the LLM text as bytes instead of characters?

yorwba 2y ago
Because if you start with characters, much of the token vocabulary would be dedicated to rare Chinese characters right off the bat. If you start from UTF-8 bytes, you can dedicate more token space to common sequences of multiple characters (i.e. words people actually use) and achieve much better compression ratios.
- RedNifre 2y ago
  I don't understand. Why would much of the vocabulary be dedicated to rare Chinese characters? Wouldn't those need to show up in the training data first? And if they did, wouldn't they also show up as weird byte sequences? And aren't UTF-8 byte sequences kinda risky for everything other than ASCII, since only ASCII bytes and header bytes are unambiguous, whereas following bytes (10***) are very ambiguous individually? I mean, sure, the LLM would notice that their meaning changes depending on preceding following- and header-bytes, but it is still not clear to me, why UTF-8 bytes are better for LLMs than characters (or even grapheme clusters). UTF-8 bytes seem like a very arbitrary choice to me. Why not do UTF-9 instead and get the most important Latin letters as single ninebitbytes?
  - yorwba 2y ago
    Yes, rare Chinese characters do show up in the training data (the rarest of them at least appear in lists of characters) and yes, they get tokenized as weird byte sequences, making the model work harder to process them, but it's better for that to happen to rare characters than to common words. It's a tradeoff. And of course UTF-8 is unlikely to be the single best encoding (e.g. Anthropic has a tokenizer that turns all caps text into a special caps lock symbol plus the regular-case equivalent) but much of it is papered over by byte-pair encoding. E.g. the most important Latin letters appear often enough that they get dedicated tokens anyways.