5 ms·

Only Windows and Java come to mind - and BOTH of those are insane for sticking to it when the entire rest of the world has moved on.

masklinn 3y ago
Windows, Java, C#, javascript, a surprising number of XML documents (though less so as time marches on thankfully), ICU I think uses UTF-16 internally (for the same historical reasons as the other 4), JOLIET file names are UCS2, some phones interpret “16-bit” SMS as UTF-16 (the spec says UCS2). > and BOTH of those are insane for sticking to it They don’t really have much of a choice because they exposed those semantics as part of the string interface (or for Windows the interaction is slow low level it can’t be hidden), they have performance guarantees and behaviours which matches that. It’s also why Python uses UTF-32, and went through the entire PEP-393 / FS complication to try and stop blowing up memory left and right: the core team considered that switching strings to UTF8 was a bridge too far. There are approximate solutions, but they come with their own costs and complications (e.g. pypy uses UTF8 strings with lazily constructed indices to emulate UTF-32 strings).
- mjevans 3y ago
  I'm not a Windows based programmer, but couldn't they leave the old API's in place, but make UTF-8 safe versions available for everyone and switch to that... E.G. with Win 11?
  - Joker_vD 3y ago
    You can set the system codepage to CP_UTF8 since Win 10, I guess, although IIRC it still doesn't work for input. But a) there is a lot of programs using A() functions that don't expect that and break in subtle ways, e.g. DBCS-encoding-aware programs suddenly break because they don't expect a codepoint to span for more than 2 bytes; b) most of the sanely written programs either use UTF-16 explicitly, or use UTF-8 internally and convert between UTF-8 and UTF-16 before/after calling W() functions.
hnfong 3y ago
I hate UTF-16 and the systems that use it with a passion, but... Windows and Java (and Javascript) adopted unicode at a time when it was thought that 64k code points would be enough for everyone. Then they prioritized backwards compatibility over anything else. Most of us have benefited from their insistence on backwards compatibility in some form or the other, so I'm really not in a position to complain about it :-/ That said, IMHO any "length" property (as opposed to `codepoints` or `bytes`) on a UTF-16 string should definitely be deprecated.

masklinn 3y ago

Windows, Java, C#, javascript, a surprising number of XML documents (though less so as time marches on thankfully), ICU I think uses UTF-16 internally (for the same historical reasons as the other 4), JOLIET file names are UCS2, some phones interpret “16-bit” SMS as UTF-16 (the spec says UCS2). > and BOTH of those are insane for sticking to it They don’t really have much of a choice because they exposed those semantics as part of the string interface (or for Windows the interaction is slow low level it can’t be hidden), they have performance guarantees and behaviours which matches that. It’s also why Python uses UTF-32, and went through the entire PEP-393 / FS complication to try and stop blowing up memory left and right: the core team considered that switching strings to UTF8 was a bridge too far. There are approximate solutions, but they come with their own costs and complications (e.g. pypy uses UTF8 strings with lazily constructed indices to emulate UTF-32 strings).

mjevans 3y ago
I'm not a Windows based programmer, but couldn't they leave the old API's in place, but make UTF-8 safe versions available for everyone and switch to that... E.G. with Win 11?
- Joker_vD 3y ago
  You can set the system codepage to CP_UTF8 since Win 10, I guess, although IIRC it still doesn't work for input. But a) there is a lot of programs using A() functions that don't expect that and break in subtle ways, e.g. DBCS-encoding-aware programs suddenly break because they don't expect a codepoint to span for more than 2 bytes; b) most of the sanely written programs either use UTF-16 explicitly, or use UTF-8 internally and convert between UTF-8 and UTF-16 before/after calling W() functions.

hnfong 3y ago

I hate UTF-16 and the systems that use it with a passion, but... Windows and Java (and Javascript) adopted unicode at a time when it was thought that 64k code points would be enough for everyone. Then they prioritized backwards compatibility over anything else. Most of us have benefited from their insistence on backwards compatibility in some form or the other, so I'm really not in a position to complain about it :-/ That said, IMHO any "length" property (as opposed to `codepoints` or `bytes`) on a UTF-16 string should definitely be deprecated.