4 ms·

Now just UTF-16 and non '\n' newline types remaining to go

by Aardwolf 1y ago

hypeatei 1y ago
UTF-16 will be quite the mountain as Windows APIs and web specifications/engines default to it for historical reasons.
- int_19h 1y ago
  It's not just Windows and JavaScript. On Apple platforms, NSString is UTF-16. On Linux, Qt uses UTF-16 strings. Looking at languages, we have Java (which is where JS got this bug from) and C# both enshrining it in their respective language specs. So it's far more pervasive than people think, and will likely be in the picture for decades to come.
  - electroly 1y ago
    ICU (International Components for Unicode, the library published by the Unicode folks) itself uses UTF-16 internally, and most of these things are built on ICU. I agree strongly with your conclusion--UTF-16 isn't going anywhere. I don't think the ICU people are even talking about changing the internals to UTF-8.
  - account42 1y ago
    Qt could really change if they wanted to and really should have by now - it's not like they keep long-term backwards compatibility anyway unlike the others that you mentioned. Of course they chose to integrate JavaScript so that's less likely now.
syncsynchalt 1y ago
Of the two UTF-16 is much less of a problem, it's trivially[1] and losslessly convertible. [1] Ok I admit, not trivially when it comes to unpaired surrogates, BOMs, endian detection, and probably a dozen other edge and corner cases I don't even know about. But you can offload the work to pretty well-understood and trouble-free library calls.
- Aardwolf 1y ago
  It causes such issues as opening of files by filename not being cross platform with standard libc functions since in Windows w-string versions are required since UTF-16 chars can have 0-bytes that aren't zero terminators
  - codedokode 1y ago
    It makes sense because UTF-16 is a sequence of uint16_t values, so both bytes must be zero, not just one.
    - syncsynchalt 1y ago
      Yes, it makes sense, but it still resulted in a lot of work. Most Unix syscalls use C-style strings, which are a string of 8-bit bytes terminated with a zero byte. With many (most?) character encodings you can continue to present string data to syscalls in the same way, since they often also reserved a byte value of zero for the same purpose. Even some multi-byte encodings would work if they chose to avoid using 0-value bytes for this reason. UTF-16LE/BE (and UTF-32 for that matter) chose not to allow for this, and the result is that if you want UTF-16 support in your existing C-string-based syscalls you need to make a second copy of every syscall which supports strings in your UTF-16 type of choice.
      - codedokode 1y ago
        > Most Unix syscalls use C-style strings, which are a string of 8-bit bytes terminated with a zero byte. With many (most?) character encodings you can continue to present string data to syscalls in the same way, since they often also reserved a byte value of zero for the same purpose That's completely wrong. If a syscall (or a function) expects text in encoding A, you should not be sending it in encoding B because it would be interpreted incorrectly, or even worse, this would become a vulnerability. For every function, encoding must be specified as are specified the types of arguments, constraints and ownership rules. Sadly many open source libraries do not do it. How are you supposed to call a function when you don't know the expected encoding? Also, it is better to send a pointer and a length of the string rather than potentially infinitely search for a zero byte. > and the result is that if you want UTF-16 support in your existing C-string-based syscalls There is no need to support multiple encodings, it only makes things complicated. The simplest solution would be to use UTF-8 for all kernel facilities as a standard. For example, it would be better if open() syscall required valid UTF-8 string for a file name. This would leave no possibility for displaying file names as question marks.
jeberle 1y ago
UTF-16 arguably is Unicode 2.0+. It's how the code point address space is defined. Code points are either 1 or 2 16-bit code units. Easy. Compare w/ UTF-8 where a code point may be 1, 2, 3, or 4 8-bit code units. UTF-16 is annoying, but it's far from the biggest design failure in Unicode.
- adgjlsfhk1 1y ago
  UTF-16 is the worst of all worlds. Either use UTF32 where code-points are fixed, or if you care about space efficiency use UTF8
  - mort96 1y ago
    UTF-32 is arguably even more worst of all worlds. You don't get fixed-size units in any meaningful way. Yes you have fixed sized code points, but those aren't the "units" you care about; you still have variable size grapheme clusters, so you still can't do things like reversing a string or splitting a string at an arbitrary index or anything else like that. Yet it consumes twice the space of UTF-16 for almost everything, and four times the space of UTF-8 for many things. UTF-32 is the worst of all worlds. UTF-16 has the teeny tiny advantage that pure Chinese text takes a bit less space in UTF-16 than UTF-8 (typically irrelevant because that advantage is outweighed by the fact that the markup surrounding the text takes more space). UTF-8 is the best option for pretty much everything. As a consequence, never use UTF-32, only use UTF-16 where necessary due to backwards compatibility, always use UTF-8 where possible.
    - kbolino 1y ago
      In order to implement grapheme cluster segmentation, you have to start with a sequence of Unicode scalars. In practice, that means a sequence of 32-bit integers, which is UTF-32 in all but name. It's not a good interchange format, but it is a necessary intermediate/internal format. There's also the problem that grapheme cluster boundaries change over time. Unicode has become a true mess.
      - 1y ago
augustk 1y ago
> Now just UTF-16 and non '\n' newline types remaining to go Also ISO 8601 (YYYY-MM-DD) should be the default date format.

Now just UTF-16 and non '\n' newline types remaining to go

hypeatei 1y ago

UTF-16 will be quite the mountain as Windows APIs and web specifications/engines default to it for historical reasons.

int_19h 1y ago
It's not just Windows and JavaScript. On Apple platforms, NSString is UTF-16. On Linux, Qt uses UTF-16 strings. Looking at languages, we have Java (which is where JS got this bug from) and C# both enshrining it in their respective language specs. So it's far more pervasive than people think, and will likely be in the picture for decades to come.
- electroly 1y ago
  ICU (International Components for Unicode, the library published by the Unicode folks) itself uses UTF-16 internally, and most of these things are built on ICU. I agree strongly with your conclusion--UTF-16 isn't going anywhere. I don't think the ICU people are even talking about changing the internals to UTF-8.
- account42 1y ago
  Qt could really change if they wanted to and really should have by now - it's not like they keep long-term backwards compatibility anyway unlike the others that you mentioned. Of course they chose to integrate JavaScript so that's less likely now.

syncsynchalt 1y ago

Of the two UTF-16 is much less of a problem, it's trivially[1] and losslessly convertible. [1] Ok I admit, not trivially when it comes to unpaired surrogates, BOMs, endian detection, and probably a dozen other edge and corner cases I don't even know about. But you can offload the work to pretty well-understood and trouble-free library calls.

Aardwolf 1y ago
It causes such issues as opening of files by filename not being cross platform with standard libc functions since in Windows w-string versions are required since UTF-16 chars can have 0-bytes that aren't zero terminators
- codedokode 1y ago
  It makes sense because UTF-16 is a sequence of uint16_t values, so both bytes must be zero, not just one.
  - syncsynchalt 1y ago
    Yes, it makes sense, but it still resulted in a lot of work. Most Unix syscalls use C-style strings, which are a string of 8-bit bytes terminated with a zero byte. With many (most?) character encodings you can continue to present string data to syscalls in the same way, since they often also reserved a byte value of zero for the same purpose. Even some multi-byte encodings would work if they chose to avoid using 0-value bytes for this reason. UTF-16LE/BE (and UTF-32 for that matter) chose not to allow for this, and the result is that if you want UTF-16 support in your existing C-string-based syscalls you need to make a second copy of every syscall which supports strings in your UTF-16 type of choice.
    - codedokode 1y ago
      > Most Unix syscalls use C-style strings, which are a string of 8-bit bytes terminated with a zero byte. With many (most?) character encodings you can continue to present string data to syscalls in the same way, since they often also reserved a byte value of zero for the same purpose That's completely wrong. If a syscall (or a function) expects text in encoding A, you should not be sending it in encoding B because it would be interpreted incorrectly, or even worse, this would become a vulnerability. For every function, encoding must be specified as are specified the types of arguments, constraints and ownership rules. Sadly many open source libraries do not do it. How are you supposed to call a function when you don't know the expected encoding? Also, it is better to send a pointer and a length of the string rather than potentially infinitely search for a zero byte. > and the result is that if you want UTF-16 support in your existing C-string-based syscalls There is no need to support multiple encodings, it only makes things complicated. The simplest solution would be to use UTF-8 for all kernel facilities as a standard. For example, it would be better if open() syscall required valid UTF-8 string for a file name. This would leave no possibility for displaying file names as question marks.

jeberle 1y ago

UTF-16 arguably is Unicode 2.0+. It's how the code point address space is defined. Code points are either 1 or 2 16-bit code units. Easy. Compare w/ UTF-8 where a code point may be 1, 2, 3, or 4 8-bit code units. UTF-16 is annoying, but it's far from the biggest design failure in Unicode.

adgjlsfhk1 1y ago
UTF-16 is the worst of all worlds. Either use UTF32 where code-points are fixed, or if you care about space efficiency use UTF8
- mort96 1y ago
  UTF-32 is arguably even more worst of all worlds. You don't get fixed-size units in any meaningful way. Yes you have fixed sized code points, but those aren't the "units" you care about; you still have variable size grapheme clusters, so you still can't do things like reversing a string or splitting a string at an arbitrary index or anything else like that. Yet it consumes twice the space of UTF-16 for almost everything, and four times the space of UTF-8 for many things. UTF-32 is the worst of all worlds. UTF-16 has the teeny tiny advantage that pure Chinese text takes a bit less space in UTF-16 than UTF-8 (typically irrelevant because that advantage is outweighed by the fact that the markup surrounding the text takes more space). UTF-8 is the best option for pretty much everything. As a consequence, never use UTF-32, only use UTF-16 where necessary due to backwards compatibility, always use UTF-8 where possible.
  - kbolino 1y ago
    In order to implement grapheme cluster segmentation, you have to start with a sequence of Unicode scalars. In practice, that means a sequence of 32-bit integers, which is UTF-32 in all but name. It's not a good interchange format, but it is a necessary intermediate/internal format. There's also the problem that grapheme cluster boundaries change over time. Unicode has become a true mess.
    - 1y ago

augustk 1y ago

> Now just UTF-16 and non '\n' newline types remaining to go Also ISO 8601 (YYYY-MM-DD) should be the default date format.