3 ms·
It's not that surprising that models encode Byzantine Music Notation characters using separate tokens for each UTF-8 byte, since they're unlikely to occur often
by yorwba 2y ago
It's not that surprising that models encode Byzantine Music Notation characters using separate tokens for each UTF-8 byte, since they're unlikely to occur often enough for byte-pair encoding to allocate dedicated multi-byte tokens to them.
What isn't clear to me is where ASCII text with 64 added to every byte occurs in the wild.
- the8472 2y ago> What isn't clear to me is where ASCII text with 64 added to every byte occurs in the wild. maybe it has some general ascii bittwiddling capability for base64 and uuencode that help here too
- immibis 2y agoHave you checked the UTF-8 encodings of all the typical "Unicode fonts" e.g. circled letters, superscripts and so on?
- csense 2y agoTranslating uppercase to lowercase adds 32. Maybe it's not "thinking" in terms of "add 64", but rather "translate uppercase to lowercase, twice".
- bmenrigh 2y agoPossibly because of xor with 0xc0 which, for lowercase ascii, has the same effect as adding 64. Probably seeing xor-encoded input is common in the training data.
- zeckalpha 2y agoNaN boxing?