7 ms·

IMHO it's completely fair to criticize others for the externalities that they create. UTF-16 is a design flaw, and Microsoft repeatedly decides to release a ne

by dlitz 13y ago

IMHO it's completely fair to criticize others for the externalities that they create. UTF-16 is a design flaw, and Microsoft repeatedly decides to release a new version of Windows that fails to meaningfully address that flaw. To date, MS hasn't communicated a vision for a future of Windows where nobody has to deal with UTF-16, we can only conclude that their vision is that for the next 100 years, the rest of us will still be paying to support their use of UTF-16. This isn't just about UTF-16 either. Think about how much IE6 has cost everyone who's made a website in the last 10 years. Microsoft's penchant for building and marketing high-friction platforms is a huge drain on innovation (and a big barrier-to-entry for people who want to learn programming), and the company deserves an enormous amount of criticism for it.

asveikau 13y ago
Can you explain how utf-16 is a design flaw? This sounds totally kooky to me. Are you thinking of the more limited UCS-2? Utf-16 represents the same set of chars as utf-8. It's true that lots of errors can occur when people assume 1 wchar = 1 codepoint = 1 glyph but utf-8 has similar complexity and I've seen plenty of people screw it up too. It sounds much more to me like you are saying that "not working how I am used to" is the same as "design flaw". Edit: put another way, the NT kernel since its inception represented and continues to represent all strings as 16 bit integers, saying that they need to "meaningfully address this" is like saying Linux should migrate away from 8-bit strings; there is no reason to do it. A lossless conversion exists and I fail to see it as a big deal, it's just a historical thing because NT's initial development predates UTF-8.
- malkia 13y ago
  You need two versions of functions. You need two modes for lots of text programs to use. It doubles the space for things like storing identifiers for programming languages, which normally would use up to 7 bits (for big percentage of the languages). UTF-8 can work with all 7-bit ASCII characters, and that's what it's great about it.
  - asveikau 13y ago
    > You need two versions of functions. For Windows this is entirely a compatibility thing. One could imagine a world in which that was not strictly necessary. A good best practice for a Win32 app is to always use 16-bit strings when calling system APIs and pretend the "ANSI" versions don't exist; I would not recommend anything else. In NT the 8-bit versions generally do nothing but convert to 16-bit and call the "real" function. In recent versions (I think Win7 was the first to do this), AFAIK the 8-bit shims typically exist in another module, so if you don't use them they don't get loaded. In NT on ARM the 8-bit shims are not even there. > It doubles the space for things like storing identifiers for programming languages Which is why GetProcAddress() still takes an 8-bit string. Just because the kernel (and hence the syscall interface) uses 16-bit everywhere doesn't mean you can't use 8-bit strings in your own process, or that you can't selectively pick what makes sense for your use. (By the way, none of what I'm saying is a criticism of UTF-8. I think it's a very clever encoding.)
    - option_greek 13y ago
      If they were using UTF-8, wouldn't just one version of function suffice ? That is, instead of using a shim, just determine which version to call based on the encoding.
    - nitrogen 13y ago
      There would be no need for the 16-bit versions if they had used UTF-8. The same function signatures would work for ASCII, ISO-8859, or UTF-8.
      - asveikau 13y ago
        NT development began before UTF-8 existed. Surely you're not suggesting that the authors should have used an encoding that did not yet exist?
- dlitz 13y ago
  Imagine a world where we don't have to convert between multiple charsets, or even think about them, beyond "text" vs "binary". That world is perfectly achievable, it's basically already happened outside the Microsoft ecosystem. OSX and Linux default to using UTF-8 everywhere, Ruby on Rails is UTF-8 only, the assumed source encoding in Python 3 is UTF-8, URI percent-encoding is done exclusively using UTF-8, HTML5 defaults to UTF-8... The list goes on and on. "UTF-8" is becoming synonymous with "text". In an all-UTF-8 world, if you want to build a URI parser, or a CSV parser, or an HTML parser, or really anything that does any kind of text processing (except rendering), you can just assume ASCII and everything will work as long as you're 8-bit clean. Even non-US codepages are all more-or-less supersets of ASCII. The only major exceptions are EBCDIC and UTF-16. Because of UTF-16, we have to build charset conversion/negotiation into every single interface boundary, where we could otherwise just be 8-bit clean except in places where text actually has to be rendered. It's completely unnecessary friction that undermines one of the big motivations for creating Unicode in the first place: coming up with a single standard for text representation, so that we don't have to deal with the mess of handling multiple charsets everywhere. None of these decisions is a big deal in isolation, but it's a death by 1000 cuts. When you have mountains of unnecessary complexity, it only serves to make programming inaccessible to ordinary people, which is how we get awful policies like SOPA, software patents, and the like.
  - asveikau 13y ago
    So much hand-wavy text in there... There's no inherent, fundamental reason in the universe other than momentum that says you've got to use an 8-bit encoding. You could serve to be a bit more honest about how arbitrary that call really is and where it comes from. In fact many higher level languages even outside the MSFT ecosystem are happily using UTF-16 everywhere natively without problems, they just do a small conversion step at a syscall to run on your beloved Unix. AFAIK the JVM is working this way for example. This "imagine a world" game can just as easily go the other way: "imagine a world in which everything is UTF-16 ...". I think it's disingenuous to say that it matters one way or the other, that your favorite is better and all others, even if they are a 1-to-1 mapping with your favorite, constitute a "design flaw". Because information theory does not care; it is just a different encoding for the same damn thing. In truth it makes little difference as long as you are consistent, and you are whining simply because not everybody picked the same thing as your favorite. Your claim that UTF-16 is a "design flaw" has not been validated, just that you don't like encodings ever to change and UTF-16 isn't your favorite. It's very hard to consider that anything other than whining. I thought we software types are supposed to be big on abstractions and coming up with clever ways of managing complexity? It seems rather rigid to say you'll only ever deal with one text encoding. Lastly even in 2013 this is a total lie: > OSX and Linux default to using UTF-8 everywhere, If that's true then how come on virtually every Unix-like system I've set up for the last ~15 years one of the first things I've had to do is edit ~/.profile to futz around with LC_CTYPE or whatever to ask for UTF-8? I am pretty sure every Unix-like system I have set up gave me Latin1 by default, even quite recently. I think you are underestimating the extent to which UTF-8 is a crude hack designed to avoid rewriting ancient C programs that did very much the wrong approach to localization. There was a time before UTF-8 existed and became popular when it was a fairly common viewpoint that proper Unicode support involved making a clean break with the old char type. You are right to say that UTF-8 "won" in most places but the fact that the NT kernel or the JVM use 16-bit chars reflects that prior history. I think a more mature attitude would be to accept this, that it came from a time and a place and is a different way of working, rather than call it "wrong".
- deleted 13y ago
  [deleted]
- dmytrish 13y ago
  Another example of UTF-16 brokenness: it is byte-order dependent, that is, programmer must care about big-endian vs little-endian encoding of 16-bit numbers. This caused another hack: using non-text BOM (Byte-Order Mark) symbols to denote what endianness is used. I think it's very ugly and may be ok for Windows-x86 worlds, but not acceptable for variety of platforms connected to the Internet. UTF-8 avoids endianness problems completely.
  - asveikau 13y ago
    A legit complaint. Though I'll say, in early Unicode before UTF-8 existed, this was probably seen as less of a problem than using the older non-Unicode charsets, and probably rightly so. Also, if you keep a clean separation between serialization and in-memory representation, the BOM hack is not such a big deal. Serialize code can write a BOM, and de-serialize can do the char swapping and omit the BOM in RAM, then in-memory it's always host byte order and without BOM. Or you can just have the on-disk/over-the-wire format be UTF-8, which is common today on platforms where UTF-16 is used in RAM. (I'll repeat a point that seems to be lost on a lot of people in this thread, that Microsoft adopted 16-bit chars for NT before UTF-8 existed, so that was not an option.)

by dlitz 13y ago

asveikau 13y ago

Can you explain how utf-16 is a design flaw? This sounds totally kooky to me. Are you thinking of the more limited UCS-2? Utf-16 represents the same set of chars as utf-8. It's true that lots of errors can occur when people assume 1 wchar = 1 codepoint = 1 glyph but utf-8 has similar complexity and I've seen plenty of people screw it up too. It sounds much more to me like you are saying that "not working how I am used to" is the same as "design flaw". Edit: put another way, the NT kernel since its inception represented and continues to represent all strings as 16 bit integers, saying that they need to "meaningfully address this" is like saying Linux should migrate away from 8-bit strings; there is no reason to do it. A lossless conversion exists and I fail to see it as a big deal, it's just a historical thing because NT's initial development predates UTF-8.

malkia 13y ago
You need two versions of functions. You need two modes for lots of text programs to use. It doubles the space for things like storing identifiers for programming languages, which normally would use up to 7 bits (for big percentage of the languages). UTF-8 can work with all 7-bit ASCII characters, and that's what it's great about it.
- asveikau 13y ago
  > You need two versions of functions. For Windows this is entirely a compatibility thing. One could imagine a world in which that was not strictly necessary. A good best practice for a Win32 app is to always use 16-bit strings when calling system APIs and pretend the "ANSI" versions don't exist; I would not recommend anything else. In NT the 8-bit versions generally do nothing but convert to 16-bit and call the "real" function. In recent versions (I think Win7 was the first to do this), AFAIK the 8-bit shims typically exist in another module, so if you don't use them they don't get loaded. In NT on ARM the 8-bit shims are not even there. > It doubles the space for things like storing identifiers for programming languages Which is why GetProcAddress() still takes an 8-bit string. Just because the kernel (and hence the syscall interface) uses 16-bit everywhere doesn't mean you can't use 8-bit strings in your own process, or that you can't selectively pick what makes sense for your use. (By the way, none of what I'm saying is a criticism of UTF-8. I think it's a very clever encoding.)
  - option_greek 13y ago
    If they were using UTF-8, wouldn't just one version of function suffice ? That is, instead of using a shim, just determine which version to call based on the encoding.
  - nitrogen 13y ago
    There would be no need for the 16-bit versions if they had used UTF-8. The same function signatures would work for ASCII, ISO-8859, or UTF-8.
    - asveikau 13y ago
      NT development began before UTF-8 existed. Surely you're not suggesting that the authors should have used an encoding that did not yet exist?
dlitz 13y ago
Imagine a world where we don't have to convert between multiple charsets, or even think about them, beyond "text" vs "binary". That world is perfectly achievable, it's basically already happened outside the Microsoft ecosystem. OSX and Linux default to using UTF-8 everywhere, Ruby on Rails is UTF-8 only, the assumed source encoding in Python 3 is UTF-8, URI percent-encoding is done exclusively using UTF-8, HTML5 defaults to UTF-8... The list goes on and on. "UTF-8" is becoming synonymous with "text". In an all-UTF-8 world, if you want to build a URI parser, or a CSV parser, or an HTML parser, or really anything that does any kind of text processing (except rendering), you can just assume ASCII and everything will work as long as you're 8-bit clean. Even non-US codepages are all more-or-less supersets of ASCII. The only major exceptions are EBCDIC and UTF-16. Because of UTF-16, we have to build charset conversion/negotiation into every single interface boundary, where we could otherwise just be 8-bit clean except in places where text actually has to be rendered. It's completely unnecessary friction that undermines one of the big motivations for creating Unicode in the first place: coming up with a single standard for text representation, so that we don't have to deal with the mess of handling multiple charsets everywhere. None of these decisions is a big deal in isolation, but it's a death by 1000 cuts. When you have mountains of unnecessary complexity, it only serves to make programming inaccessible to ordinary people, which is how we get awful policies like SOPA, software patents, and the like.
- asveikau 13y ago
  So much hand-wavy text in there... There's no inherent, fundamental reason in the universe other than momentum that says you've got to use an 8-bit encoding. You could serve to be a bit more honest about how arbitrary that call really is and where it comes from. In fact many higher level languages even outside the MSFT ecosystem are happily using UTF-16 everywhere natively without problems, they just do a small conversion step at a syscall to run on your beloved Unix. AFAIK the JVM is working this way for example. This "imagine a world" game can just as easily go the other way: "imagine a world in which everything is UTF-16 ...". I think it's disingenuous to say that it matters one way or the other, that your favorite is better and all others, even if they are a 1-to-1 mapping with your favorite, constitute a "design flaw". Because information theory does not care; it is just a different encoding for the same damn thing. In truth it makes little difference as long as you are consistent, and you are whining simply because not everybody picked the same thing as your favorite. Your claim that UTF-16 is a "design flaw" has not been validated, just that you don't like encodings ever to change and UTF-16 isn't your favorite. It's very hard to consider that anything other than whining. I thought we software types are supposed to be big on abstractions and coming up with clever ways of managing complexity? It seems rather rigid to say you'll only ever deal with one text encoding. Lastly even in 2013 this is a total lie: > OSX and Linux default to using UTF-8 everywhere, If that's true then how come on virtually every Unix-like system I've set up for the last ~15 years one of the first things I've had to do is edit ~/.profile to futz around with LC_CTYPE or whatever to ask for UTF-8? I am pretty sure every Unix-like system I have set up gave me Latin1 by default, even quite recently. I think you are underestimating the extent to which UTF-8 is a crude hack designed to avoid rewriting ancient C programs that did very much the wrong approach to localization. There was a time before UTF-8 existed and became popular when it was a fairly common viewpoint that proper Unicode support involved making a clean break with the old char type. You are right to say that UTF-8 "won" in most places but the fact that the NT kernel or the JVM use 16-bit chars reflects that prior history. I think a more mature attitude would be to accept this, that it came from a time and a place and is a different way of working, rather than call it "wrong".
deleted 13y ago
[deleted]
dmytrish 13y ago
Another example of UTF-16 brokenness: it is byte-order dependent, that is, programmer must care about big-endian vs little-endian encoding of 16-bit numbers. This caused another hack: using non-text BOM (Byte-Order Mark) symbols to denote what endianness is used. I think it's very ugly and may be ok for Windows-x86 worlds, but not acceptable for variety of platforms connected to the Internet. UTF-8 avoids endianness problems completely.
- asveikau 13y ago
  A legit complaint. Though I'll say, in early Unicode before UTF-8 existed, this was probably seen as less of a problem than using the older non-Unicode charsets, and probably rightly so. Also, if you keep a clean separation between serialization and in-memory representation, the BOM hack is not such a big deal. Serialize code can write a BOM, and de-serialize can do the char swapping and omit the BOM in RAM, then in-memory it's always host byte order and without BOM. Or you can just have the on-disk/over-the-wire format be UTF-8, which is common today on platforms where UTF-16 is used in RAM. (I'll repeat a point that seems to be lost on a lot of people in this thread, that Microsoft adopted 16-bit chars for NT before UTF-8 existed, so that was not an option.)