7 ms·

The letter à can be represented in two different ways in Unicode: * As the single code point U+00E0, which encodes 'à'. * Or as the sequence of two code point

The letter à can be represented in two different ways in Unicode: * As the single code point U+00E0, which encodes 'à'. * Or as the sequence of two code points U+0061 U+0300, which respectively encode the Latin letter 'a' and the combining acute accent (it's hard to display as a standalone character, so go e.g. here: https://www.compart.com/fr/unicode/U+0300 https://www.compart.com/fr/unicode/U+0300). These two code points get combined into a single grapheme cluster, the technical name for what most people consider to be a character, that displays as 'à'. As you can see, there is no difference in the visual representation of à and à. But if you inspect the string (in python or whatever) then you'll see that one of them has one code point (one "char"), while the second has two. If you're on Windows, an easy way is to type it into pwsh: PS> 'à'.EnumerateRunes() | % { "U+{0:X4}" -f $_.Value } U+0061 U+0300 PS> 'à'.EnumerateRunes() | % { "U+{0:X4}" -f $_.Value } U+00E0

alganet 4y ago
The character "à" as a single byte E0, as far as I know, is not UTF-8. This is the 224th character of the extended ASCII table (also ISO 8859). It's another system: https://en.wikipedia.org/wiki/Extended_ASCII https://en.wikipedia.org/wiki/Extended_ASCII This representation is incompatible with UTF-8, which marks the octet series Cx, Dx, Ex and Fx as leading bytes. You can see the table here: https://en.wikipedia.org/wiki/UTF-8#Codepage_layout https://en.wikipedia.org/wiki/UTF-8#Codepage_layout What I believe Kernighan _wants_, is to reuse a lot of code that was designed for ASCII. Code that treats 1 char as 1 byte. In order to do that, he is going to encode each unicode character using 4 of whatever type he was previously using before, zeroing the "pad" bytes, which is exactly what UTF-32 does. This way, he doesn't have to fundamentally change all the types he is already working with. (ps: I looked at the commit after I wrote this, he doesn't seem to be doing what I suggested). Your PS snippet failed for me "Method invocation failed because [System.String] does not contain a method named 'EnumerateRunes'." Dit it in bash, which supports UTF-8 and lets you explicitly set the encoding: $ LC_ALL=C bash -c 'x=à; echo ${#x}' 2 $ LC_ALL=en_US.UTF-8 bash -c 'x=à; echo ${#x}' 1 These snippets count the number of chars according to the selected encoding. If I had UTF-32 locales in my machine, "à" would appear as 4. Unfortunatelly I don't know any system that implements UTF-32 to display its behavior. Let's now reconstruct "à" in UTF-8, telling bash to use only ASCII: $ LC_ALL=C bash -c 'printf %b \\xC3\\xA0' à I've tunneled UTF-8 through an ASCII program, and because my terminal accepts UTF-8, everything is fine. Didn't even had to use \u escapes. This backwards compatibility is by design, and chosing between UTF-8, 16 and 32 plays a role in how you're going to deal with these byte sequences.
- bzxcvbn 4y ago
  > The character "à" as a single byte E0, as far as I know, is not UTF-8. It's not, but I wrote "Unicode" and "code point", not "UTF-8" and "code unit". It sounds like you don't know the difference, which makes this whole discussion a bit silly and probably a bit pointless. One of the best explanations I've read is this one: https://docs.microsoft.com/en-us/dotnet/standard/base-types/character-encoding-introduction https://docs.microsoft.com/en-us/dotnet/standard/base-types/... and especially the section on grapheme clusters. > Your PS snippet failed for me "Method invocation failed because [System.String] does not contain a method named 'EnumerateRunes'." Sounds like you're using the outdated windows powershell, not pwsh.
  - alganet 4y ago
    I'm talking about how many bytes it takes to store characters in these encodings, and how the lengh of these sequences changes according to each standard. You are talking about another level of abstraction. These UTF-16 builtins are first-class citizens in .NET, and you never have to deal with the raw structure behind all of it unless you want to. C does not have these primitives, a string there is composed of smokes and mirrors using byte arrays. In summary, you are talking about using a complete, black box Unicode implementation. I'm talking about the tradeoffs one might encounter when crafting stuff that goes inside one of these black boxes, and it's different from the black box you know.
    - bzxcvbn 4y ago
      Okay, let's back up a little. You were replying about a comment that said: > Yes, every codepoint is 32 bits, but every character can be made up of one or more codepoints. And you expressed surprise and incredulity at the idea. I explained that a "character" (which, given the context, is to be taken as a "grapheme") can, indeed, take up more than one code point. This has nothing to do with UTF-32, UTF-8, or .NET's implementation. It's inherent to how Unicode works. As an even more obvious example, try to figure out how many bytes are in the character ȃ encoded as UTF-32, using whatever method you'd like. (I say "more obvious" because you'll be forced to copy-paste the character I've used in your code editor.) Other examples include emojis such as this one: https://emojipedia.org/woman-golfing/ https://emojipedia.org/woman-golfing/ (that I can't use in the comment box for some reason). Then for some reason you felt compelled to offer a rebuttal that showed you didn't understand the difference between a grapheme and a code point. So I just linked some doc. I'm not attacking you.

alganet 4y ago

The character "à" as a single byte E0, as far as I know, is not UTF-8. This is the 224th character of the extended ASCII table (also ISO 8859). It's another system: https://en.wikipedia.org/wiki/Extended_ASCII https://en.wikipedia.org/wiki/Extended_ASCII This representation is incompatible with UTF-8, which marks the octet series Cx, Dx, Ex and Fx as leading bytes. You can see the table here: https://en.wikipedia.org/wiki/UTF-8#Codepage_layout https://en.wikipedia.org/wiki/UTF-8#Codepage_layout What I believe Kernighan _wants_, is to reuse a lot of code that was designed for ASCII. Code that treats 1 char as 1 byte. In order to do that, he is going to encode each unicode character using 4 of whatever type he was previously using before, zeroing the "pad" bytes, which is exactly what UTF-32 does. This way, he doesn't have to fundamentally change all the types he is already working with. (ps: I looked at the commit after I wrote this, he doesn't seem to be doing what I suggested). Your PS snippet failed for me "Method invocation failed because [System.String] does not contain a method named 'EnumerateRunes'." Dit it in bash, which supports UTF-8 and lets you explicitly set the encoding: $ LC_ALL=C bash -c 'x=à; echo ${#x}' 2 $ LC_ALL=en_US.UTF-8 bash -c 'x=à; echo ${#x}' 1 These snippets count the number of chars according to the selected encoding. If I had UTF-32 locales in my machine, "à" would appear as 4. Unfortunatelly I don't know any system that implements UTF-32 to display its behavior. Let's now reconstruct "à" in UTF-8, telling bash to use only ASCII: $ LC_ALL=C bash -c 'printf %b \\xC3\\xA0' à I've tunneled UTF-8 through an ASCII program, and because my terminal accepts UTF-8, everything is fine. Didn't even had to use \u escapes. This backwards compatibility is by design, and chosing between UTF-8, 16 and 32 plays a role in how you're going to deal with these byte sequences.

bzxcvbn 4y ago
> The character "à" as a single byte E0, as far as I know, is not UTF-8. It's not, but I wrote "Unicode" and "code point", not "UTF-8" and "code unit". It sounds like you don't know the difference, which makes this whole discussion a bit silly and probably a bit pointless. One of the best explanations I've read is this one: https://docs.microsoft.com/en-us/dotnet/standard/base-types/character-encoding-introduction https://docs.microsoft.com/en-us/dotnet/standard/base-types/... and especially the section on grapheme clusters. > Your PS snippet failed for me "Method invocation failed because [System.String] does not contain a method named 'EnumerateRunes'." Sounds like you're using the outdated windows powershell, not pwsh.
- alganet 4y ago
  I'm talking about how many bytes it takes to store characters in these encodings, and how the lengh of these sequences changes according to each standard. You are talking about another level of abstraction. These UTF-16 builtins are first-class citizens in .NET, and you never have to deal with the raw structure behind all of it unless you want to. C does not have these primitives, a string there is composed of smokes and mirrors using byte arrays. In summary, you are talking about using a complete, black box Unicode implementation. I'm talking about the tradeoffs one might encounter when crafting stuff that goes inside one of these black boxes, and it's different from the black box you know.
  - bzxcvbn 4y ago
    Okay, let's back up a little. You were replying about a comment that said: > Yes, every codepoint is 32 bits, but every character can be made up of one or more codepoints. And you expressed surprise and incredulity at the idea. I explained that a "character" (which, given the context, is to be taken as a "grapheme") can, indeed, take up more than one code point. This has nothing to do with UTF-32, UTF-8, or .NET's implementation. It's inherent to how Unicode works. As an even more obvious example, try to figure out how many bytes are in the character ȃ encoded as UTF-32, using whatever method you'd like. (I say "more obvious" because you'll be forced to copy-paste the character I've used in your code editor.) Other examples include emojis such as this one: https://emojipedia.org/woman-golfing/ https://emojipedia.org/woman-golfing/ (that I can't use in the comment box for some reason). Then for some reason you felt compelled to offer a rebuttal that showed you didn't understand the difference between a grapheme and a code point. So I just linked some doc. I'm not attacking you.