3 ms·
I still wonder how the models picked up the semantic mapping between Unicode tags and ordinary ASCII characters. The mapping is written in the Unicode specs, y
by xg15 2y ago
I still wonder how the models picked up the semantic mapping between Unicode tags and ordinary ASCII characters. The mapping is written in the Unicode specs, yes, but there is nothing in the actual bytes of a tag that indicates the corresponding ASCII character.
I'm also not aware there are large text corpora written in tag characters - actually, I'd be surprised if there is any prose text at all: The characters don't show up in any browser or text editor, they are not officially used for anything and even the two former intended uses were restricted to country codes, not actual sentences.
How did they even go through preprocessing? How is the tokenization dictionary and input embedding constructed for characters that are never used anywhere?
- goodside 2y ago(I’m the person interviewed in the article.) The trick is Unicode code points are only assigned individual tokens if they’re nontrivially used outside of some other already tokenized sequence, and Unicode tag block code points are only ever used in flag emojis. Unused or rarely used Unicode code points are given a fallback encoding that just encodes the numerical code point value in two special tokens. Because the Unicode tag block is by design the first 128 chars in ASCII repeated, the second token of the tokenized output directly corresponds to the ASCII value of the character.
- xg15 2y agoAh, so the model "sees" the tags as literal ASCII characters interspersed with special tokens? That would make more sense.
- goodside 2y agoMore or less; they’re not literally the same tokens as “a”, “b”, “c” but I’d speculate the mapping is learned from some other examples of ASCII (or just Roman letters) being repeated in other obscure parts of Unicode — Gothic glyphs, bubble letters, etc. Once the model has seen enough ASCII represented as Unicode code points whose tokenizations alternate between meaningless and meaningful (e.g. “~l~i~k~e~ ~t~h~i~s”) it learns how to read it regardless of what the ”~” is.
- theamk 2y agoThose invisible letters have codepoints of ASCII letters + 0xE0000. For example compare "U+E0054 TAG LATIN CAPITAL LETTER T"[0] vs "U+0054 LATIN CAPITAL LETTER T"[1] A simple assumption of "codepoint is 16 bit" will be enough to decode. You can see this in python: >>> x = '(copy message from article here)' >>> x 'https://wuzzi.net/copirate/\U000e0001\U000e0054\U000e0068\U000e0065\U000e0020\U000e0073\U000e0061\U000e006c\U000e0065\U000e0073\U000e0020\U000e0066\U000e006f\U000e0072\U000e0020\U000e0053\U000e0065\U000e0061\U000e0074\U000e0074\U000e006c\U000e0065\U000e0020\U000e0077\U000e0065\U000e0072\U000e0065\U000e0020\U000e0055\U000e0053\U000e0044\U000e0020\U000e0031\U000e0032\U000e0030\U000e0030\U000e0030\U000e0030\U000e007f,' >>> "".join([chr(ord(c) & 0xFFFF) for c in x]) 'https://wuzzi.net/copirate/\x01The sales for Seattle were USD 120000\x7f,' maybe authors worked with Windows or Java too much? :) I always thought wchar's were a horrible idea. [0] https://www.fileformat.info/info/unicode/char/e0054/index.htm https://www.fileformat.info/info/unicode/char/e0054/index.ht... [1] https://www.fileformat.info/info/unicode/char/54/index.htm https://www.fileformat.info/info/unicode/char/54/index.htm
- xg15 2y agoWasn't aware that the byte representation aligns so directly with the ASCII letters. Thanks a lot for the info.