8 ms·

People who don't understand surrogate pairs and think that UTF-16 == UCS-2 tend to argue that UTF-16 is better because (in their minds) it's a fixed-sized chara

People who don't understand surrogate pairs and think that UTF-16 == UCS-2 tend to argue that UTF-16 is better because (in their minds) it's a fixed-sized character encoding (it's not). Even UTF-32 is not a fixed-sized character encoding!! It's a fixed-sized codepoint encoding. Characters can be composed of more than one codepoint. Even if you think "hey, I'll use pre-composed codepoints", you'll fail because not every legitimate, canonically decomposed character has a single-codepoint pre-composition. The disadvantages to UTF-16 far outweigh its one advantage (the one you mentioned). (There is one more advantage to UTF-16, and it's that Win32 uses it all over the place. But that's not much of an advantage, and Windows does seem to be making progress towards putting UTF-8 on an equal footing (if not better).)

ivanbakel 8y ago
Does Win32 use UTF-16 or WTF-16?
- vardump 8y ago
  For path and object names, it's the latter. WTF-16 "standard" is just UTF-16 that allows invalid surrogate pairs. Windows kernel style handling. It's a shame you were voted down for a perfectly legitimate (if misleading) question. This is a very common gotcha most developers ignore until they encounter their first invalid surrogate pair file path... UTF-16 can't represent all possible Windows file names or paths.
  - ivanbakel 8y ago
    Was it misleading? I was just trying to understand if Win32 used WTF-16 everywhere (and some strings mostly happened to be valid UTF-16), or if there was some place where it used UTF instead.
    - vardump 8y ago
      I think object names and paths are WTF-16, that's where NT kernel is involved. Note that Microsoft doesn't call it that way, but it's a useful naming distinction regardless. Applications can of course use whatever they please. Just need to be careful with the possible invalid object or file name surrogate pairs, not to fail on those and to pass them with alteration. Well, misleading in that sense you did get the downvotes... That said I find WTF-16 pretty appropriate name for what Windows does internally on the kernel level.
      - cryptonector 8y ago
        In general kernels know nothing about codesets, but filesystems do (e.g., ZFS definitely does). I call this just-use-16 and just-use-8 depending on whether the strings are arrays of bytes or of uint16_t's.
    - Izkata 8y ago
      I'll bet they saw it and thought you were just being insulting (what-the-fuck-16), but it apparently stands for wobbly-transformation-format. Had to look this up myself to be sure.
xenadu02 8y ago
This is a historical oddity. At the time Apple and Microsoft (plus a few others) game up with Unicode the assumption was that we’d be able to fit everything into 16 bits. That turned out not to be true but binary compatibility made switching to UTF-8 impossible.
- eridius 8y ago
  Not just binary compatibility, also API compatibility. All of the performance guarantees of NSString are based around the internal representation being UTF-16 (or ASCII). Similarly -[NSString length] is defined as the number of UTF-16 code units, and all of the subscripting / range operations are defined in terms of UTF-16 code units.
  - int_19h 8y ago
    Same thing goes for Java and .NET. It's somewhat ironic that those who were the earliest on the Unicode bandwagon ended up saddled with all this compatibility garbage...
    - eridius 8y ago
      What gets me is that this affects later languages too. For example, Swift's internal String storage is either ASCII or UTF-16, precisely because it needs to maintain NSString's performance guarantees when bridging to Obj-C. That said, it seems the plan for Swift 5 is to change String's internal storage to be UTF-8, and then when bridging to NSString it will provide amortized constant-time UTF-16 codepoint access by calculating and caching "breadcrumbs" for fast lookup of UTF-16 offsets.
WorldMaker 8y ago
Plus while UTF-32 is currently defined as equivalent UCS-4, the surrogate pairs tricks of UTF-8/UTF-16 are still theoretically possible even if they don't currently encode anything. Not that we have a reason today to expect there to be a plane bigger than the Astral Plane, but in the 90s no one expected the Astral Plane to open up. Anyone picking UCS-4 today and pretending its UTF-32 but not prepared to handle surrogate pairs for the Currently Imaginary Plane beyond, is potentially making the exact same mistakes of UCS-2 early adopters and will have the same UTF-16/WTF-16 headaches should that day arrive.
- cryptonector 8y ago
  I expect UTF-16 to die eventually and let us have more than 21 bits of codepoints, but we'll never ever need even 30 bits, let alone 32 bits, of codepoints. I have to imagine that you're just jesting :0
  - WorldMaker 8y ago
    It's not entirely a jest, but it is funny to think about. The question will be then, as it has been every time before: what do we mean by "Universal" in Unicode? One very obvious jest is that at the current rate of emoji expansion it may be inevitable to open up the next plane just for emoji. Less of a jest is that there's always more symbols to encode. We've been a wonderfully creative species, and we're still rediscovering old written languages and discovering new bits of physics and math that can benefit from new symbols. We've got tons of interesting fictional language encodings to consider that are currently not likely to be encoded in Unicode today because they are locked in that struggle between the needs of Academia, the interests the Enthusiasts, and the money/complications of Intellectual Property laws. (Klingon, Tolkien's Elvish languages, D'ni, Timelord Sigils, etc.) But if those seesaws tip towards enthusiasts or academia that changes quickly. Meanwhile, creative artists keep inventing new glyphs and languages to encode. That's just our one species on this planet. What if we discover more or get better at encoding existing non-human languages? Then we really get to test that "Universal" bit in the name, huh? (That's also obviously a bit of jest, but an interesting sci-fi hope/yearning there, too. It would be nice to encode some non-human languages for a change.) So, all of the above, based on current assumptions (emojis will slow down, IP law remains vicious, Fermi's Paradox won't been solved in our lifetimes, etc) may still be unlikely to push us past the Astral Plane, but that doesn't make it silly to assume that we may never need to open the next plane.
avar 8y ago
In practice most applications that require a chars(str) function can get away with returning the wrong result for things outside the BMP, as opposed to UTF-8 where you need to start caring as soon as you hit words like "café". Even if you you do require chars(str) for large strings outside the BMP those characters were so rare before emojis that you could waste a single bit on "contains any non-BMP?" and almost always do the work in O(1) time as opposed to O(n) for UTF-8.
- cryptonector 8y ago
  Sorry, that just generates garbage when dealing with things outside the BMP. That can be a lot more common than you think. E.g., when dealing with Chinese characters in a context where unification is not welcomed (e.g., in China).
  - avar 8y ago
    Yes, you're right that it generates garbage, but that's besides the point. The point is that a huge number of programmers, especially in the 90s and early 00s would argue for UTF-16 on the basis of it being a fixed width encoding in practice. Maybe they didn't know that it actually wasn't, or they knew and didn't care because they never had to deal with anything outside the BMP. The overlap between Windows programmers producing software for e.g. in the U.S. or European market and those that would have ever encountered a non-BMP used to be tiny until emojis came along. So yes, while not in theory, in practice you could get away with treating UTF-16 like fixed width encoding like UCS-2 for a huge number of applications where you could reap the benefits of constant-time chars(str) and charoffset(str, N).
    - cryptonector 8y ago
      The garbage is super annoying. Please stop. Human scripts are O(N), too bad. You can build indices (must, for large documents), but you can't really avoid this being O(N). And we're not even talking about normalization. People get upset about these things and blame Unicode, but the problems are not with Unicode -- they are semantics problems with our scripts that Unicode deals with about as well as can be hoped for. The only thing I'd remove from Unicode is pre-compositions and the associated normal forms NFC and NFKC. But note that that wouldn't remove the need for normalization.

ivanbakel 8y ago

Does Win32 use UTF-16 or WTF-16?

vardump 8y ago
For path and object names, it's the latter. WTF-16 "standard" is just UTF-16 that allows invalid surrogate pairs. Windows kernel style handling. It's a shame you were voted down for a perfectly legitimate (if misleading) question. This is a very common gotcha most developers ignore until they encounter their first invalid surrogate pair file path... UTF-16 can't represent all possible Windows file names or paths.
- ivanbakel 8y ago
  Was it misleading? I was just trying to understand if Win32 used WTF-16 everywhere (and some strings mostly happened to be valid UTF-16), or if there was some place where it used UTF instead.
  - vardump 8y ago
    I think object names and paths are WTF-16, that's where NT kernel is involved. Note that Microsoft doesn't call it that way, but it's a useful naming distinction regardless. Applications can of course use whatever they please. Just need to be careful with the possible invalid object or file name surrogate pairs, not to fail on those and to pass them with alteration. Well, misleading in that sense you did get the downvotes... That said I find WTF-16 pretty appropriate name for what Windows does internally on the kernel level.
    - cryptonector 8y ago
      In general kernels know nothing about codesets, but filesystems do (e.g., ZFS definitely does). I call this just-use-16 and just-use-8 depending on whether the strings are arrays of bytes or of uint16_t's.
  - Izkata 8y ago
    I'll bet they saw it and thought you were just being insulting (what-the-fuck-16), but it apparently stands for wobbly-transformation-format. Had to look this up myself to be sure.

xenadu02 8y ago

This is a historical oddity. At the time Apple and Microsoft (plus a few others) game up with Unicode the assumption was that we’d be able to fit everything into 16 bits. That turned out not to be true but binary compatibility made switching to UTF-8 impossible.

eridius 8y ago
Not just binary compatibility, also API compatibility. All of the performance guarantees of NSString are based around the internal representation being UTF-16 (or ASCII). Similarly -[NSString length] is defined as the number of UTF-16 code units, and all of the subscripting / range operations are defined in terms of UTF-16 code units.
- int_19h 8y ago
  Same thing goes for Java and .NET. It's somewhat ironic that those who were the earliest on the Unicode bandwagon ended up saddled with all this compatibility garbage...
  - eridius 8y ago
    What gets me is that this affects later languages too. For example, Swift's internal String storage is either ASCII or UTF-16, precisely because it needs to maintain NSString's performance guarantees when bridging to Obj-C. That said, it seems the plan for Swift 5 is to change String's internal storage to be UTF-8, and then when bridging to NSString it will provide amortized constant-time UTF-16 codepoint access by calculating and caching "breadcrumbs" for fast lookup of UTF-16 offsets.

WorldMaker 8y ago

Plus while UTF-32 is currently defined as equivalent UCS-4, the surrogate pairs tricks of UTF-8/UTF-16 are still theoretically possible even if they don't currently encode anything. Not that we have a reason today to expect there to be a plane bigger than the Astral Plane, but in the 90s no one expected the Astral Plane to open up. Anyone picking UCS-4 today and pretending its UTF-32 but not prepared to handle surrogate pairs for the Currently Imaginary Plane beyond, is potentially making the exact same mistakes of UCS-2 early adopters and will have the same UTF-16/WTF-16 headaches should that day arrive.

cryptonector 8y ago
I expect UTF-16 to die eventually and let us have more than 21 bits of codepoints, but we'll never ever need even 30 bits, let alone 32 bits, of codepoints. I have to imagine that you're just jesting :0
- WorldMaker 8y ago
  It's not entirely a jest, but it is funny to think about. The question will be then, as it has been every time before: what do we mean by "Universal" in Unicode? One very obvious jest is that at the current rate of emoji expansion it may be inevitable to open up the next plane just for emoji. Less of a jest is that there's always more symbols to encode. We've been a wonderfully creative species, and we're still rediscovering old written languages and discovering new bits of physics and math that can benefit from new symbols. We've got tons of interesting fictional language encodings to consider that are currently not likely to be encoded in Unicode today because they are locked in that struggle between the needs of Academia, the interests the Enthusiasts, and the money/complications of Intellectual Property laws. (Klingon, Tolkien's Elvish languages, D'ni, Timelord Sigils, etc.) But if those seesaws tip towards enthusiasts or academia that changes quickly. Meanwhile, creative artists keep inventing new glyphs and languages to encode. That's just our one species on this planet. What if we discover more or get better at encoding existing non-human languages? Then we really get to test that "Universal" bit in the name, huh? (That's also obviously a bit of jest, but an interesting sci-fi hope/yearning there, too. It would be nice to encode some non-human languages for a change.) So, all of the above, based on current assumptions (emojis will slow down, IP law remains vicious, Fermi's Paradox won't been solved in our lifetimes, etc) may still be unlikely to push us past the Astral Plane, but that doesn't make it silly to assume that we may never need to open the next plane.

avar 8y ago

In practice most applications that require a chars(str) function can get away with returning the wrong result for things outside the BMP, as opposed to UTF-8 where you need to start caring as soon as you hit words like "café". Even if you you do require chars(str) for large strings outside the BMP those characters were so rare before emojis that you could waste a single bit on "contains any non-BMP?" and almost always do the work in O(1) time as opposed to O(n) for UTF-8.

cryptonector 8y ago
Sorry, that just generates garbage when dealing with things outside the BMP. That can be a lot more common than you think. E.g., when dealing with Chinese characters in a context where unification is not welcomed (e.g., in China).
- avar 8y ago
  Yes, you're right that it generates garbage, but that's besides the point. The point is that a huge number of programmers, especially in the 90s and early 00s would argue for UTF-16 on the basis of it being a fixed width encoding in practice. Maybe they didn't know that it actually wasn't, or they knew and didn't care because they never had to deal with anything outside the BMP. The overlap between Windows programmers producing software for e.g. in the U.S. or European market and those that would have ever encountered a non-BMP used to be tiny until emojis came along. So yes, while not in theory, in practice you could get away with treating UTF-16 like fixed width encoding like UCS-2 for a huge number of applications where you could reap the benefits of constant-time chars(str) and charoffset(str, N).
  - cryptonector 8y ago
    The garbage is super annoying. Please stop. Human scripts are O(N), too bad. You can build indices (must, for large documents), but you can't really avoid this being O(N). And we're not even talking about normalization. People get upset about these things and blame Unicode, but the problems are not with Unicode -- they are semantics problems with our scripts that Unicode deals with about as well as can be hoped for. The only thing I'd remove from Unicode is pre-compositions and the associated normal forms NFC and NFKC. But note that that wouldn't remove the need for normalization.