11 ms·

So this all justifies my general stance of "find a Unicode expert" when questions of unicode come up. But that's pretty wilfully blind (although practical). I

So this all justifies my general stance of "find a Unicode expert" when questions of unicode come up. But that's pretty wilfully blind (although practical). I would like to understand why UTF-32 didn't catch on as The Standard Unicode for the modern world. it seems that - albeit memory wasteful - it would sidestep a lot of these issues.

layer8 4y ago
UTF-32 doesn’t completely solve this either, because grapheme clusters.
- pnathan 4y ago
  oh, _bother_. :-/
- bsder 4y ago
  And endianness.
  - layer8 4y ago
    Since Unicode only uses 21 bits, it would be possible to define an endianness-safe 32-bit encoding. E.g. shift left by one and set the LSB.
  - a1369209993 4y ago
    To be fair, UTF-32 is actually much better about this than most other encodings for which endianness issues are even a thing. The top byte of any UTF-32 code unit has to zero, and NUL isn't a valid (ie text) character, so any ASCII characters anywhere in the file make the endianness unambiguous. More generally, every code unit in the file has to have the form 00xxyy00, and the possible values for xx and yy must be in the range 0 thru 16, so there are only 17*17-1 = 288 unicode code points that can possibly occur in a endianness-ambiguous string/file. And out of those, you have confusions like "Ā" (U+0100 LATIN CAPITAL LETTER A WITH MACRON) versus "𐀀" (U+10000 LINEAR B SYLLABLE B008 A) or "𠀀" (U+20000) versus "Ȁ" (U+200), most of which can only happen if you have no idea what language you're expecting. (Some, like "𐄀" (U+10100 AEGEAN WORD SEPARATOR LINE), are the same regardless.) Whereas here's a valid bash script: ⌀℀ ⼀戀椀渀⼀戀愀猀栀਀猀甀搀漀椀搀਀⌀ 2>/dev/null || true echo "Hello, World!" # ਀ (You can almost do this with C/C++ as well, but something needs to #define "⼀⼀" as "int" for it to work properly, and then only if the compiler accepts non-ascii characters (namely "⼀" aka U+2F00 KANGXI RADICAL ONE) in identifiers.)
jffry 4y ago
> why UTF-32 didn't catch on > memory wasteful The answer is in the question really. If you've got a big pile of mostly-ascii data, quadrupling memory/storage to encode it as UTF32 is going to be a pretty tough sell
- Shish2k 4y ago
  How many "big piles of mostly-ascii data" are there though? (Does anyone want to write a script which searches /dev/mem and categorises pages of RAM into ascii-or-not-ascii so we can get some meaningful numbers? :P ) (If you’re doing number-crunching on giant CSVs, maybe I can see it being important, but all the ascii files on my desktop that I can think of are pretty trivial)
  - steveklabnik 4y ago
    Pretty much every website, due to HTML being ASCII.
    - ravi-delia 4y ago
      I feel like compression will do a better job than deciding on an encoding scheme ahead of time, no? Once gziped I wouldn't expect a difference
      - steveklabnik 4y ago
        I thought I'd read something about it, but when I googled, what I did find was this old HN comment: https://news.ycombinator.com/item?id=8514519 https://news.ycombinator.com/item?id=8514519 > UTF-8 + gzip is 32% smaller than UTF-32 + gzip using the HN frontpage as corpus.
      - JoshTriplett 4y ago
        You still have to decompress it on the other end, to actually parse and use it. At which point you have four times the memory usage, unless you turn it into some smaller in-memory encoding...such as UTF-8.
harikb 4y ago
UTF-8 has no upper limit on the number of possible characters / emoji's, now or in the future. Everything else has. And then is UTF-16 which has all the pains of UTF-8 with none of the advantages of UTF-32
- doubleunplussed 4y ago
  That's not quite right. UTF-8 is not arbitrary length. Officially, it's at most four bytes, of which 21 bits are usable for encoding codepoints - so that's an upper limit of 2^21 codepoints. There is an initial byte encoding the length as a series of ones, so if you went ahead and extended the standard to simply allow more bytes, you could get up to 8 bytes, of which 48 bits would be usable. I can see that a six-byte version with 31 data bits was previously standardised before they settled on four. I guess you could extend it further by allowing more than one initial byte encoding the length, then it would be arbitrary length. But at that point I'm not sure if it loses its self-synchronising ability, and in any case it would be a different standard at that point.
  - bhaney 4y ago
    > if you went ahead and extended the standard to simply allow more bytes, you could get up to 8 bytes I think you'd only be able to go up to 7, since 10xxxxxx is still reserved for trailing octets. And even with 7, the entire first octet is consumed by the length indicator alone. So you get 0xxxxxxx, 110xxxxx, 1110xxxx, 11110xxx, 111110xx, 1111110x, and 11111110 as the 7 different length-indicating head octets. In the last case, you'd have 36 usable bits for encoding a codepoint.
    - doubleunplussed 4y ago
      Ah, I forgot 10xxxxxx was not usable, but I also forgot 0xxxxxxx was. What about 11111111? If that's valid then it's 8, if I'm thinking straight.
      - bhaney 4y ago
        11111111 is technically possible to use, but it would cause some problems. Sending it over the wire would break telnet, for example. Also since we already introduced 11111110 for 7-byte encodes, we're getting dangerously close to making the UTF-16 BOM character (11111111 11111110) accidentally show up in UTF-8 (this is also why 11111110 wasn't in the original maximum-6-byte UTF-8 spec). I still don't think it's possible to have the UTF-16 BOM show up in our hypothetical extended UTF-8, since 11111111 could never be immediately followed by 11111110 (or vice versa) in a well-formed UTF-8 stream. Also note that if you did add 11111111 as a valid head octet representing an 8 octet long encoding, you'd still only have 42 usable bits (since the first byte is still entirely consumed by the length indicator)
jcranmer 4y ago
One of the root problems here is that a concept like "character" is extremely underdefined. What you want to do in terms of counting the memory a string will take up, indicating a particular spot in the string, mapping to arrow keys or the backspace key in a visual program, knowing how far to indent to drop a caret in a visual error report, knowing the width of the string in a visual (especially fixed-width) text display. For ASCII text, you can use the same value to represent all possible slightly different definitions, but in non-ASCII, you need to use different definitions, and there's no one true definition that solves all use cases (no, not even grapheme clusters).
qalmakka 4y ago
It's pointless to have char32_t if you still need to pull several megabytes of ICU to normalize the string first in order to remove characters spanning over multiple codepoints. UTF32 is arguably dangerous because of this, it's yet another attempt to replicate ASCII but with Unicode. The only sane encoding out there is UTF-8, and that's it. If you have to always assume your string is not really splittable without a library, you won't do dangerous stuff such as assuming `wcslen(L"menù") == 4`.
userbinator 4y ago
This all justifies my stance of "don't add complexity unless you absolutely need it." People are surprised, confused, and sometimes even offended by the fact that I do almost all my work with a plain text editor and a terminal. I have the same sentiment towards those who insist upon large complex fragile stacks of tools and then wonder why they spend so much time chasing down bugs in those rather than working on what they actually intended to.
- iudqnolq 4y ago
  This justifies my stance of "find the most normal setup and use it". This is a bug in the less popular third-party lsp package for emacs, which is already quite unpopular. I use VSCode, an enormously complex system. But so many other people use it there tends not to be this sort of bug. And in the rare case there is one I just wait a few days until someone else solves it.
  - c-cube 4y ago
    In this particular case, Vscode doesn't have the bug... Because it's responsible for this mess in the first place. LSP was first introduced in Vscode which is written in C++ and typescript, so they found it convenient to assume a UTF-16 encoding (as is native to Javascript). If LSP had originated elsewhere it'd probably use codepoint or UTF-8 byte offsets from the start.
    - iudqnolq 4y ago
      Exactly. On a moral level that's obviously bad. But on a practical level that supports my decision to use VSCode - they got to define the standard such that everyone else has to work around their mess, making themselves have less problems.
sebzim4500 4y ago
Isn't the problem here that both UTF-8 and UTF-16 were being used at once, with incorrect conversion between offsets. I don't see how adding another encoding would help here. Sure if, everyone used UTF-32 for everything then these problems would go away but they would also go away if everyone used UTF-8, and most uncompressed files would be 4 times smaller.

layer8 4y ago

UTF-32 doesn’t completely solve this either, because grapheme clusters.

pnathan 4y ago
oh, _bother_. :-/
bsder 4y ago
And endianness.
- layer8 4y ago
  Since Unicode only uses 21 bits, it would be possible to define an endianness-safe 32-bit encoding. E.g. shift left by one and set the LSB.
- a1369209993 4y ago
  To be fair, UTF-32 is actually much better about this than most other encodings for which endianness issues are even a thing. The top byte of any UTF-32 code unit has to zero, and NUL isn't a valid (ie text) character, so any ASCII characters anywhere in the file make the endianness unambiguous. More generally, every code unit in the file has to have the form 00xxyy00, and the possible values for xx and yy must be in the range 0 thru 16, so there are only 17*17-1 = 288 unicode code points that can possibly occur in a endianness-ambiguous string/file. And out of those, you have confusions like "Ā" (U+0100 LATIN CAPITAL LETTER A WITH MACRON) versus "𐀀" (U+10000 LINEAR B SYLLABLE B008 A) or "𠀀" (U+20000) versus "Ȁ" (U+200), most of which can only happen if you have no idea what language you're expecting. (Some, like "𐄀" (U+10100 AEGEAN WORD SEPARATOR LINE), are the same regardless.) Whereas here's a valid bash script: ⌀℀ ⼀戀椀渀⼀戀愀猀栀਀猀甀搀漀椀搀਀⌀ 2>/dev/null || true echo "Hello, World!" # ਀ (You can almost do this with C/C++ as well, but something needs to #define "⼀⼀" as "int" for it to work properly, and then only if the compiler accepts non-ascii characters (namely "⼀" aka U+2F00 KANGXI RADICAL ONE) in identifiers.)

jffry 4y ago

> why UTF-32 didn't catch on > memory wasteful The answer is in the question really. If you've got a big pile of mostly-ascii data, quadrupling memory/storage to encode it as UTF32 is going to be a pretty tough sell

Shish2k 4y ago
How many "big piles of mostly-ascii data" are there though? (Does anyone want to write a script which searches /dev/mem and categorises pages of RAM into ascii-or-not-ascii so we can get some meaningful numbers? :P ) (If you’re doing number-crunching on giant CSVs, maybe I can see it being important, but all the ascii files on my desktop that I can think of are pretty trivial)
- steveklabnik 4y ago
  Pretty much every website, due to HTML being ASCII.
  - ravi-delia 4y ago
    I feel like compression will do a better job than deciding on an encoding scheme ahead of time, no? Once gziped I wouldn't expect a difference
    - steveklabnik 4y ago
      I thought I'd read something about it, but when I googled, what I did find was this old HN comment: https://news.ycombinator.com/item?id=8514519 https://news.ycombinator.com/item?id=8514519 > UTF-8 + gzip is 32% smaller than UTF-32 + gzip using the HN frontpage as corpus.
    - JoshTriplett 4y ago
      You still have to decompress it on the other end, to actually parse and use it. At which point you have four times the memory usage, unless you turn it into some smaller in-memory encoding...such as UTF-8.

harikb 4y ago

UTF-8 has no upper limit on the number of possible characters / emoji's, now or in the future. Everything else has. And then is UTF-16 which has all the pains of UTF-8 with none of the advantages of UTF-32

doubleunplussed 4y ago
That's not quite right. UTF-8 is not arbitrary length. Officially, it's at most four bytes, of which 21 bits are usable for encoding codepoints - so that's an upper limit of 2^21 codepoints. There is an initial byte encoding the length as a series of ones, so if you went ahead and extended the standard to simply allow more bytes, you could get up to 8 bytes, of which 48 bits would be usable. I can see that a six-byte version with 31 data bits was previously standardised before they settled on four. I guess you could extend it further by allowing more than one initial byte encoding the length, then it would be arbitrary length. But at that point I'm not sure if it loses its self-synchronising ability, and in any case it would be a different standard at that point.
- bhaney 4y ago
  > if you went ahead and extended the standard to simply allow more bytes, you could get up to 8 bytes I think you'd only be able to go up to 7, since 10xxxxxx is still reserved for trailing octets. And even with 7, the entire first octet is consumed by the length indicator alone. So you get 0xxxxxxx, 110xxxxx, 1110xxxx, 11110xxx, 111110xx, 1111110x, and 11111110 as the 7 different length-indicating head octets. In the last case, you'd have 36 usable bits for encoding a codepoint.
  - doubleunplussed 4y ago
    Ah, I forgot 10xxxxxx was not usable, but I also forgot 0xxxxxxx was. What about 11111111? If that's valid then it's 8, if I'm thinking straight.
    - bhaney 4y ago
      11111111 is technically possible to use, but it would cause some problems. Sending it over the wire would break telnet, for example. Also since we already introduced 11111110 for 7-byte encodes, we're getting dangerously close to making the UTF-16 BOM character (11111111 11111110) accidentally show up in UTF-8 (this is also why 11111110 wasn't in the original maximum-6-byte UTF-8 spec). I still don't think it's possible to have the UTF-16 BOM show up in our hypothetical extended UTF-8, since 11111111 could never be immediately followed by 11111110 (or vice versa) in a well-formed UTF-8 stream. Also note that if you did add 11111111 as a valid head octet representing an 8 octet long encoding, you'd still only have 42 usable bits (since the first byte is still entirely consumed by the length indicator)

jcranmer 4y ago

One of the root problems here is that a concept like "character" is extremely underdefined. What you want to do in terms of counting the memory a string will take up, indicating a particular spot in the string, mapping to arrow keys or the backspace key in a visual program, knowing how far to indent to drop a caret in a visual error report, knowing the width of the string in a visual (especially fixed-width) text display. For ASCII text, you can use the same value to represent all possible slightly different definitions, but in non-ASCII, you need to use different definitions, and there's no one true definition that solves all use cases (no, not even grapheme clusters).

qalmakka 4y ago

It's pointless to have char32_t if you still need to pull several megabytes of ICU to normalize the string first in order to remove characters spanning over multiple codepoints. UTF32 is arguably dangerous because of this, it's yet another attempt to replicate ASCII but with Unicode. The only sane encoding out there is UTF-8, and that's it. If you have to always assume your string is not really splittable without a library, you won't do dangerous stuff such as assuming `wcslen(L"menù") == 4`.

userbinator 4y ago

This all justifies my stance of "don't add complexity unless you absolutely need it." People are surprised, confused, and sometimes even offended by the fact that I do almost all my work with a plain text editor and a terminal. I have the same sentiment towards those who insist upon large complex fragile stacks of tools and then wonder why they spend so much time chasing down bugs in those rather than working on what they actually intended to.

iudqnolq 4y ago
This justifies my stance of "find the most normal setup and use it". This is a bug in the less popular third-party lsp package for emacs, which is already quite unpopular. I use VSCode, an enormously complex system. But so many other people use it there tends not to be this sort of bug. And in the rare case there is one I just wait a few days until someone else solves it.
- c-cube 4y ago
  In this particular case, Vscode doesn't have the bug... Because it's responsible for this mess in the first place. LSP was first introduced in Vscode which is written in C++ and typescript, so they found it convenient to assume a UTF-16 encoding (as is native to Javascript). If LSP had originated elsewhere it'd probably use codepoint or UTF-8 byte offsets from the start.
  - iudqnolq 4y ago
    Exactly. On a moral level that's obviously bad. But on a practical level that supports my decision to use VSCode - they got to define the standard such that everyone else has to work around their mess, making themselves have less problems.

sebzim4500 4y ago

Isn't the problem here that both UTF-8 and UTF-16 were being used at once, with incorrect conversion between offsets. I don't see how adding another encoding would help here. Sure if, everyone used UTF-32 for everything then these problems would go away but they would also go away if everyone used UTF-8, and most uncompressed files would be 4 times smaller.