3 ms·

To make the point more clear, the female golfing Unicode “character” is encoded as follows in various UTFs: UTF16 (12 bytes total): \ud83c\udfcc\ufe0f\u200d\u

To make the point more clear, the female golfing Unicode “character” is encoded as follows in various UTFs: UTF16 (12 bytes total): \ud83c\udfcc\ufe0f\u200d\u2640\ufe0f UTF32 (20 bytes total): u+0001f3ccu+0000fe0fu+0000200du+00002640u+0000fe0f UTF8 (16 bytes total): \xf0\x9f\x8f\x8c\xef\xb8\x8f\xe2\x80\x8d\xe2\x99\x80\xef\xb8\x8f

alganet 4y ago
You are still presenting me with abstractly encoded data. \u and u+ are in a higher level of abstraction. The only raw bytes I am seeing here are in the UTF-8 string, which you decided to serialize as hexadecimals (did you had a choice? why?). If you had all of these expressed as pure hexadecimals (or octals, or any single-byte unit), how would they be serialized? Then, once all of them are just hexadecimals, how would you go about parsing one of these sequences of hexadecimals into _characters_? (each hexa representing a raw byte, just like in the awk codebase we are talking about) Another question: do you need first to parse the raw bytes into characters before recognizing graphemes, or can you do it all at once for both variable-length and fixed-length encodings?
- jsjohnst 4y ago
  > You are still presenting me with abstractly encoded data. That’s the actual encoding for that grapheme as specified by the spec for UTF8, UTF16, and UTF32. > \u and u+ are in a higher level of abstraction. No, it’s not, it’s how you write escaped 16bit and 32bit hexadecimal strings for UTF-16 and UTF-32 respectively. Notice there’s 4 hex characters after \u and 8 hex after u+. Those are the “raw bytes” in hex. > The only raw bytes I am seeing here are in the UTF-8 string, which you decided to serialize as hexadecimals All three forms are “raw bytes” in hex form. \x is how you represent an escaped 8 bit UTF-8 byte in hex. > Another question: do you need first to parse the raw bytes into characters before recognizing graphemes, or can you do it all at once for both variable-length and fixed-length encodings? You need to “parse” (more like read for UTF16 and UTF32, as there’s not much actual parsing outside byte order handling) the raw bytes into codepoints. To try to minimize confusion, consider Character and Grapheme as synonymous. They are made up of one or more codepoints. It really doesn’t matter if it’s variable length or fixed length, you still have to get the codepoints before you can determine character/graphemes.
  - alganet 4y ago
    I am asking you "how do you purify water"? And you're holding a bottle of Fiji and telling me "look, it's simple". You're absolutely right about what is a character and what is a grapheme. I already said that, this subject is done. You're right, no need to come back to it. You win, I already yielded several comments ago. Now, to the other subject: I would very much prefer if we talked only about bytes. Yes, talking only about bytes makes things harder. Somewhere down the line, there must be an implementation that deals with the byte sequence. I'm talking about these, just above the assembler (for awk). There IS confusion at this level, no way to avoid it except abstracting it by yourself, byte by byte (or, 4 bytes by 4 bytes in UTF-32).

alganet 4y ago

You are still presenting me with abstractly encoded data. \u and u+ are in a higher level of abstraction. The only raw bytes I am seeing here are in the UTF-8 string, which you decided to serialize as hexadecimals (did you had a choice? why?). If you had all of these expressed as pure hexadecimals (or octals, or any single-byte unit), how would they be serialized? Then, once all of them are just hexadecimals, how would you go about parsing one of these sequences of hexadecimals into _characters_? (each hexa representing a raw byte, just like in the awk codebase we are talking about) Another question: do you need first to parse the raw bytes into characters before recognizing graphemes, or can you do it all at once for both variable-length and fixed-length encodings?

jsjohnst 4y ago
> You are still presenting me with abstractly encoded data. That’s the actual encoding for that grapheme as specified by the spec for UTF8, UTF16, and UTF32. > \u and u+ are in a higher level of abstraction. No, it’s not, it’s how you write escaped 16bit and 32bit hexadecimal strings for UTF-16 and UTF-32 respectively. Notice there’s 4 hex characters after \u and 8 hex after u+. Those are the “raw bytes” in hex. > The only raw bytes I am seeing here are in the UTF-8 string, which you decided to serialize as hexadecimals All three forms are “raw bytes” in hex form. \x is how you represent an escaped 8 bit UTF-8 byte in hex. > Another question: do you need first to parse the raw bytes into characters before recognizing graphemes, or can you do it all at once for both variable-length and fixed-length encodings? You need to “parse” (more like read for UTF16 and UTF32, as there’s not much actual parsing outside byte order handling) the raw bytes into codepoints. To try to minimize confusion, consider Character and Grapheme as synonymous. They are made up of one or more codepoints. It really doesn’t matter if it’s variable length or fixed length, you still have to get the codepoints before you can determine character/graphemes.
- alganet 4y ago
  I am asking you "how do you purify water"? And you're holding a bottle of Fiji and telling me "look, it's simple". You're absolutely right about what is a character and what is a grapheme. I already said that, this subject is done. You're right, no need to come back to it. You win, I already yielded several comments ago. Now, to the other subject: I would very much prefer if we talked only about bytes. Yes, talking only about bytes makes things harder. Somewhere down the line, there must be an implementation that deals with the byte sequence. I'm talking about these, just above the assembler (for awk). There IS confusion at this level, no way to avoid it except abstracting it by yourself, byte by byte (or, 4 bytes by 4 bytes in UTF-32).