7 ms·

> Porting that from ASCII to UTF-32 is just dividing the length in bytes by 4. For UTF-8, you'd have to iterate over each character and figure out if there is a

> Porting that from ASCII to UTF-32 is just dividing the length in bytes by 4. For UTF-8, you'd have to iterate over each character and figure out if there is a combination of bytes that collapse into a single character. You have to do that with UTF-32 as well. Yes, every codepoint is 32 bits, but every character can be made up of one or more codepoints.

alganet 4y ago
What? U+004D in UTF-8 is 4D, in UTF-32 is 0000004D U+0430 in UTF-8 is D0 B0, in UTF-32 is 00000430 I took this examples from page 102 of The Unicode Standard Version 5.0. These leading zeros on the UTF-32 are what is omitted from UTF-8. One of the encodings is optimized for space, the other for processing speed. I am no expert on unicode though, there might be something I'm missing (ligatures, this kind of stuff). I would gladly accept a more in depth expalanation on why I am wrong on this.
- bzxcvbn 4y ago
  The letter à can be represented in two different ways in Unicode: * As the single code point U+00E0, which encodes 'à'. * Or as the sequence of two code points U+0061 U+0300, which respectively encode the Latin letter 'a' and the combining acute accent (it's hard to display as a standalone character, so go e.g. here: https://www.compart.com/fr/unicode/U+0300 https://www.compart.com/fr/unicode/U+0300). These two code points get combined into a single grapheme cluster, the technical name for what most people consider to be a character, that displays as 'à'. As you can see, there is no difference in the visual representation of à and à. But if you inspect the string (in python or whatever) then you'll see that one of them has one code point (one "char"), while the second has two. If you're on Windows, an easy way is to type it into pwsh: PS> 'à'.EnumerateRunes() | % { "U+{0:X4}" -f $_.Value } U+0061 U+0300 PS> 'à'.EnumerateRunes() | % { "U+{0:X4}" -f $_.Value } U+00E0
  - alganet 4y ago
    The character "à" as a single byte E0, as far as I know, is not UTF-8. This is the 224th character of the extended ASCII table (also ISO 8859). It's another system: https://en.wikipedia.org/wiki/Extended_ASCII https://en.wikipedia.org/wiki/Extended_ASCII This representation is incompatible with UTF-8, which marks the octet series Cx, Dx, Ex and Fx as leading bytes. You can see the table here: https://en.wikipedia.org/wiki/UTF-8#Codepage_layout https://en.wikipedia.org/wiki/UTF-8#Codepage_layout What I believe Kernighan _wants_, is to reuse a lot of code that was designed for ASCII. Code that treats 1 char as 1 byte. In order to do that, he is going to encode each unicode character using 4 of whatever type he was previously using before, zeroing the "pad" bytes, which is exactly what UTF-32 does. This way, he doesn't have to fundamentally change all the types he is already working with. (ps: I looked at the commit after I wrote this, he doesn't seem to be doing what I suggested). Your PS snippet failed for me "Method invocation failed because [System.String] does not contain a method named 'EnumerateRunes'." Dit it in bash, which supports UTF-8 and lets you explicitly set the encoding: $ LC_ALL=C bash -c 'x=à; echo ${#x}' 2 $ LC_ALL=en_US.UTF-8 bash -c 'x=à; echo ${#x}' 1 These snippets count the number of chars according to the selected encoding. If I had UTF-32 locales in my machine, "à" would appear as 4. Unfortunatelly I don't know any system that implements UTF-32 to display its behavior. Let's now reconstruct "à" in UTF-8, telling bash to use only ASCII: $ LC_ALL=C bash -c 'printf %b \\xC3\\xA0' à I've tunneled UTF-8 through an ASCII program, and because my terminal accepts UTF-8, everything is fine. Didn't even had to use \u escapes. This backwards compatibility is by design, and chosing between UTF-8, 16 and 32 plays a role in how you're going to deal with these byte sequences.
    - bzxcvbn 4y ago
      > The character "à" as a single byte E0, as far as I know, is not UTF-8. It's not, but I wrote "Unicode" and "code point", not "UTF-8" and "code unit". It sounds like you don't know the difference, which makes this whole discussion a bit silly and probably a bit pointless. One of the best explanations I've read is this one: https://docs.microsoft.com/en-us/dotnet/standard/base-types/character-encoding-introduction https://docs.microsoft.com/en-us/dotnet/standard/base-types/... and especially the section on grapheme clusters. > Your PS snippet failed for me "Method invocation failed because [System.String] does not contain a method named 'EnumerateRunes'." Sounds like you're using the outdated windows powershell, not pwsh.

alganet 4y ago

What? U+004D in UTF-8 is 4D, in UTF-32 is 0000004D U+0430 in UTF-8 is D0 B0, in UTF-32 is 00000430 I took this examples from page 102 of The Unicode Standard Version 5.0. These leading zeros on the UTF-32 are what is omitted from UTF-8. One of the encodings is optimized for space, the other for processing speed. I am no expert on unicode though, there might be something I'm missing (ligatures, this kind of stuff). I would gladly accept a more in depth expalanation on why I am wrong on this.

bzxcvbn 4y ago
The letter à can be represented in two different ways in Unicode: * As the single code point U+00E0, which encodes 'à'. * Or as the sequence of two code points U+0061 U+0300, which respectively encode the Latin letter 'a' and the combining acute accent (it's hard to display as a standalone character, so go e.g. here: https://www.compart.com/fr/unicode/U+0300 https://www.compart.com/fr/unicode/U+0300). These two code points get combined into a single grapheme cluster, the technical name for what most people consider to be a character, that displays as 'à'. As you can see, there is no difference in the visual representation of à and à. But if you inspect the string (in python or whatever) then you'll see that one of them has one code point (one "char"), while the second has two. If you're on Windows, an easy way is to type it into pwsh: PS> 'à'.EnumerateRunes() | % { "U+{0:X4}" -f $_.Value } U+0061 U+0300 PS> 'à'.EnumerateRunes() | % { "U+{0:X4}" -f $_.Value } U+00E0
- alganet 4y ago
  The character "à" as a single byte E0, as far as I know, is not UTF-8. This is the 224th character of the extended ASCII table (also ISO 8859). It's another system: https://en.wikipedia.org/wiki/Extended_ASCII https://en.wikipedia.org/wiki/Extended_ASCII This representation is incompatible with UTF-8, which marks the octet series Cx, Dx, Ex and Fx as leading bytes. You can see the table here: https://en.wikipedia.org/wiki/UTF-8#Codepage_layout https://en.wikipedia.org/wiki/UTF-8#Codepage_layout What I believe Kernighan _wants_, is to reuse a lot of code that was designed for ASCII. Code that treats 1 char as 1 byte. In order to do that, he is going to encode each unicode character using 4 of whatever type he was previously using before, zeroing the "pad" bytes, which is exactly what UTF-32 does. This way, he doesn't have to fundamentally change all the types he is already working with. (ps: I looked at the commit after I wrote this, he doesn't seem to be doing what I suggested). Your PS snippet failed for me "Method invocation failed because [System.String] does not contain a method named 'EnumerateRunes'." Dit it in bash, which supports UTF-8 and lets you explicitly set the encoding: $ LC_ALL=C bash -c 'x=à; echo ${#x}' 2 $ LC_ALL=en_US.UTF-8 bash -c 'x=à; echo ${#x}' 1 These snippets count the number of chars according to the selected encoding. If I had UTF-32 locales in my machine, "à" would appear as 4. Unfortunatelly I don't know any system that implements UTF-32 to display its behavior. Let's now reconstruct "à" in UTF-8, telling bash to use only ASCII: $ LC_ALL=C bash -c 'printf %b \\xC3\\xA0' à I've tunneled UTF-8 through an ASCII program, and because my terminal accepts UTF-8, everything is fine. Didn't even had to use \u escapes. This backwards compatibility is by design, and chosing between UTF-8, 16 and 32 plays a role in how you're going to deal with these byte sequences.
  - bzxcvbn 4y ago
    > The character "à" as a single byte E0, as far as I know, is not UTF-8. It's not, but I wrote "Unicode" and "code point", not "UTF-8" and "code unit". It sounds like you don't know the difference, which makes this whole discussion a bit silly and probably a bit pointless. One of the best explanations I've read is this one: https://docs.microsoft.com/en-us/dotnet/standard/base-types/character-encoding-introduction https://docs.microsoft.com/en-us/dotnet/standard/base-types/... and especially the section on grapheme clusters. > Your PS snippet failed for me "Method invocation failed because [System.String] does not contain a method named 'EnumerateRunes'." Sounds like you're using the outdated windows powershell, not pwsh.