5 ms·

> The character "à" as a single byte E0, as far as I know, is not UTF-8. It's not, but I wrote "Unicode" and "code point", not "UTF-8" and "code unit". It soun

> The character "à" as a single byte E0, as far as I know, is not UTF-8. It's not, but I wrote "Unicode" and "code point", not "UTF-8" and "code unit". It sounds like you don't know the difference, which makes this whole discussion a bit silly and probably a bit pointless. One of the best explanations I've read is this one: https://docs.microsoft.com/en-us/dotnet/standard/base-types/character-encoding-introduction https://docs.microsoft.com/en-us/dotnet/standard/base-types/... and especially the section on grapheme clusters. > Your PS snippet failed for me "Method invocation failed because [System.String] does not contain a method named 'EnumerateRunes'." Sounds like you're using the outdated windows powershell, not pwsh.

alganet 4y ago
I'm talking about how many bytes it takes to store characters in these encodings, and how the lengh of these sequences changes according to each standard. You are talking about another level of abstraction. These UTF-16 builtins are first-class citizens in .NET, and you never have to deal with the raw structure behind all of it unless you want to. C does not have these primitives, a string there is composed of smokes and mirrors using byte arrays. In summary, you are talking about using a complete, black box Unicode implementation. I'm talking about the tradeoffs one might encounter when crafting stuff that goes inside one of these black boxes, and it's different from the black box you know.
- bzxcvbn 4y ago
  Okay, let's back up a little. You were replying about a comment that said: > Yes, every codepoint is 32 bits, but every character can be made up of one or more codepoints. And you expressed surprise and incredulity at the idea. I explained that a "character" (which, given the context, is to be taken as a "grapheme") can, indeed, take up more than one code point. This has nothing to do with UTF-32, UTF-8, or .NET's implementation. It's inherent to how Unicode works. As an even more obvious example, try to figure out how many bytes are in the character ȃ encoded as UTF-32, using whatever method you'd like. (I say "more obvious" because you'll be forced to copy-paste the character I've used in your code editor.) Other examples include emojis such as this one: https://emojipedia.org/woman-golfing/ https://emojipedia.org/woman-golfing/ (that I can't use in the comment box for some reason). Then for some reason you felt compelled to offer a rebuttal that showed you didn't understand the difference between a grapheme and a code point. So I just linked some doc. I'm not attacking you.
  - alganet 4y ago
    That reply is not my first interaction on the thread. I guess you didn't see the previous one, so perhaps you are missing context. Here's the link: https://news.ycombinator.com/item?id=32535499 https://news.ycombinator.com/item?id=32535499 "ȃ" uses two bytes in UTF-8, 2 bytes in UTF-16 and 4 bytes in UTF-32. The woman golfing uses 4 bytes in UTF-8, UTF-16 and UTF-32. My surprise was at the idea of having to look up a variable length sequence of bytes in a string stored as UTF-32. To me, UTF-32 exists exactly for the use case of "I don't want to calculate variable length" (optimizing for speed), and UTF-8 exists for "I don't want to use more bytes than I need" (optimizing for space). I never used the term "codepoint" in this entire thread until now. I was originally talking about storing bytes, hence my little discomfort of not being able to communicate what I'm trying to say. Not feeling attacked, just incapable of expressing, which might have come off wrong. Sorry about that. BTW, there are many things that I don't understand about Unicode. I never read the full standard. I don't know enough about a grapheme to tell whether it is something that impacts or not the number of bytes when storing in UTF-32. Why we are talking about UTF-8 versus UTF-32 and not Unicode as a whole? Because there is a code comment about these two in the commit linked by the OP, which sparked this particular subthread.
    - bzxcvbn 4y ago
      > The woman golfing uses 4 bytes in UTF-8, UTF-16 and UTF-32. No. It's made up of 5 code points. Each of these takes 32 bits, or 4 bytes, in UTF32. So that emoji, which is a single grapheme, uses 20 bytes in UTF32. Once again, just try it yourself, encode a string containing that single emoji in UTF32 using your favorite programming language and count the bytes!
      - alganet 4y ago
        Let's go to the source: https://www.unicode.org/faq/utf_bom.html https://www.unicode.org/faq/utf_bom.html > Q: Should I use UTF-32 (or UCS-4) for storing Unicode strings in memory? > > This depends. It may seem compelling to use UTF-32 as your internal string format because it uses one code unit per code point. (...) It also confirms what I said about implementation vs interface (or storage vs access, whatever): > Q: How about using UTF-32 interfaces in my APIs? > > Except in some environments that store text as UTF-32 in memory, most Unicode APIs are using UTF-16. (...) While a UTF-32 representation does make the programming model somewhat simpler, the increased average storage size has real drawbacks, making a complete transition to UTF-32 less compelling Finally: > Q: What is UTF-32? > > Any Unicode character can be represented as a single 32-bit unit in UTF-32. This single 4 code unit corresponds to the Unicode scalar value, which is the abstract number associated with a Unicode character. So, it does in fact reduce the complexity of implementing it _for storage_, as I suspected. And there is a tradeoff, as I mentioned. And the Unicode documentation explicitly separates the interface from the storage side of things. That's good enough for me. I mentioned before there might be some edge cases like ligatures, and you came up with a zero-width joiner example. None of this changes these fundamental properties of UTF-32 though.

alganet 4y ago

I'm talking about how many bytes it takes to store characters in these encodings, and how the lengh of these sequences changes according to each standard. You are talking about another level of abstraction. These UTF-16 builtins are first-class citizens in .NET, and you never have to deal with the raw structure behind all of it unless you want to. C does not have these primitives, a string there is composed of smokes and mirrors using byte arrays. In summary, you are talking about using a complete, black box Unicode implementation. I'm talking about the tradeoffs one might encounter when crafting stuff that goes inside one of these black boxes, and it's different from the black box you know.

bzxcvbn 4y ago
Okay, let's back up a little. You were replying about a comment that said: > Yes, every codepoint is 32 bits, but every character can be made up of one or more codepoints. And you expressed surprise and incredulity at the idea. I explained that a "character" (which, given the context, is to be taken as a "grapheme") can, indeed, take up more than one code point. This has nothing to do with UTF-32, UTF-8, or .NET's implementation. It's inherent to how Unicode works. As an even more obvious example, try to figure out how many bytes are in the character ȃ encoded as UTF-32, using whatever method you'd like. (I say "more obvious" because you'll be forced to copy-paste the character I've used in your code editor.) Other examples include emojis such as this one: https://emojipedia.org/woman-golfing/ https://emojipedia.org/woman-golfing/ (that I can't use in the comment box for some reason). Then for some reason you felt compelled to offer a rebuttal that showed you didn't understand the difference between a grapheme and a code point. So I just linked some doc. I'm not attacking you.
- alganet 4y ago
  That reply is not my first interaction on the thread. I guess you didn't see the previous one, so perhaps you are missing context. Here's the link: https://news.ycombinator.com/item?id=32535499 https://news.ycombinator.com/item?id=32535499 "ȃ" uses two bytes in UTF-8, 2 bytes in UTF-16 and 4 bytes in UTF-32. The woman golfing uses 4 bytes in UTF-8, UTF-16 and UTF-32. My surprise was at the idea of having to look up a variable length sequence of bytes in a string stored as UTF-32. To me, UTF-32 exists exactly for the use case of "I don't want to calculate variable length" (optimizing for speed), and UTF-8 exists for "I don't want to use more bytes than I need" (optimizing for space). I never used the term "codepoint" in this entire thread until now. I was originally talking about storing bytes, hence my little discomfort of not being able to communicate what I'm trying to say. Not feeling attacked, just incapable of expressing, which might have come off wrong. Sorry about that. BTW, there are many things that I don't understand about Unicode. I never read the full standard. I don't know enough about a grapheme to tell whether it is something that impacts or not the number of bytes when storing in UTF-32. Why we are talking about UTF-8 versus UTF-32 and not Unicode as a whole? Because there is a code comment about these two in the commit linked by the OP, which sparked this particular subthread.
  - bzxcvbn 4y ago
    > The woman golfing uses 4 bytes in UTF-8, UTF-16 and UTF-32. No. It's made up of 5 code points. Each of these takes 32 bits, or 4 bytes, in UTF32. So that emoji, which is a single grapheme, uses 20 bytes in UTF32. Once again, just try it yourself, encode a string containing that single emoji in UTF32 using your favorite programming language and count the bytes!
    - alganet 4y ago
      Let's go to the source: https://www.unicode.org/faq/utf_bom.html https://www.unicode.org/faq/utf_bom.html > Q: Should I use UTF-32 (or UCS-4) for storing Unicode strings in memory? > > This depends. It may seem compelling to use UTF-32 as your internal string format because it uses one code unit per code point. (...) It also confirms what I said about implementation vs interface (or storage vs access, whatever): > Q: How about using UTF-32 interfaces in my APIs? > > Except in some environments that store text as UTF-32 in memory, most Unicode APIs are using UTF-16. (...) While a UTF-32 representation does make the programming model somewhat simpler, the increased average storage size has real drawbacks, making a complete transition to UTF-32 less compelling Finally: > Q: What is UTF-32? > > Any Unicode character can be represented as a single 32-bit unit in UTF-32. This single 4 code unit corresponds to the Unicode scalar value, which is the abstract number associated with a Unicode character. So, it does in fact reduce the complexity of implementing it _for storage_, as I suspected. And there is a tradeoff, as I mentioned. And the Unicode documentation explicitly separates the interface from the storage side of things. That's good enough for me. I mentioned before there might be some edge cases like ligatures, and you came up with a zero-width joiner example. None of this changes these fundamental properties of UTF-32 though.