5 ms·

C11 has the uchar.h header (say, http://www.cplusplus.com/reference/cuchar/ http://www.cplusplus.com/reference/cuchar/ ), but it wasn't in Visual Studio last I

C11 has the uchar.h header (say, http://www.cplusplus.com/reference/cuchar/ http://www.cplusplus.com/reference/cuchar/ ), but it wasn't in Visual Studio last I checked. Most programmers will tell you to use UTF-8 for in-memory strings. But it's not easy to figure out if a particular char* is already encoded as UTF-8, and it's common for people to forget that Unicode characters can take up to 6 bytes in UTF-8. I know I'm in the minority, but I prefer to use UTF-16 or UTF-32, because nobody makes the kind of mistakes with UTF-16/UTF-32 that they make with UTF-8. Plus, you can't accidentally pass a UTF-16-encoded string to a non-Unicode-aware function.

syncsynchalt 12y ago
Unicode code points can only take 4 bytes in UTF-8, this has been true ever since the planes were capped at U+10FFFF. Or did you mean UTF-8 encoded surrogate pairs?
- maxlybbert 12y ago
  Thanks for the correction. I don't know where I originally got the "up top six bytes" number from, but I've been using it for a while. Apparently, it's out of date (looking at the original proposal, https://en.wikipedia.org/wiki/UTF-8#Description https://en.wikipedia.org/wiki/UTF-8#Description , some code points were expected to need 6 bytes, but as you say, that's no longer true).
  - syncsynchalt 12y ago
    It was true until 2003, so in this case reality has been retconned on you.
nly 12y ago
UTF-16 has surrogate pairs as well. It's the worst of both worlds.
- maxlybbert 12y ago
  I prefer UTF-32, but I'm very lonely. The benefit with UTF-16 is that you can't accidentally pass a string to, say, strlen(). But, yes, people will forget that not all Unicode code points fit in 16 bits, and won't test with the right kind of input to find out that they've done it wrong. So errors can still creep in; but I will continue to argue that those errors are less common because the fact that you're dealing with Unicode (and not ASCII or something else) is more obvious and you're more likely to need library calls (that will do the right thing) to do anything useful.
  - masklinn 12y ago
    > I prefer UTF-32, but I'm very lonely. I'm not sure why you'd prefer UTF-32, it doesn't make correct text manipulations any easier, but — much like UTF-16 — it does make incorrect assumptions and text manipulations much easier.
    - maxlybbert 12y ago
      UTF-32 means each code point takes up one char32_t. There are no surrogate pairs. However, it is true that you still have to deal with combining characters, normalization, etc. So it doesn't solve all of your problems, but it does avoid simple goofiness ( http://blog.coverity.com/2014/04/09/why-utf-16/ http://blog.coverity.com/2014/04/09/why-utf-16/ ). You still have to use libraries to do anything non-trivial.
      - masklinn 12y ago
        Right. So while UTF-16 is a terrible encoding and there is very little reason to use it, UTF-32 is little better. Use UTF-8.
        maxlybbert 12y ago
        Like I said, I'm very lonely. My only consolation is that every project I've seen that says "use UTF-8 and be careful not to do goofy things (like running a string through the encoder twice, or passing a string to old C string functions that aren't Unicode aware, or assuming each character takes one byte), without exception, constantly finds those goofy mistakes in their code. Being careful is not enough. UTF-8 for strings on disk, yes. But UTF-8 for in-memory strings has a long track record of being much harder to actually do correctly.
mpyne 12y ago
Beyond the issue noted with UTF-16 permitting surrogate pairs, it's also important to understand that UTF-16 does not free you from normalization form concerns. For instance, even within the 16-bit Basic Multilingual Plane, it is not safe to reverse a UTF-16 string by simply reversing each block of 2 bytes, as a Unicode glyph (e.g. á) can be composed of a pre-combined code point which only takes 16-bits, or a base character (a) followed by a combiner (´). The pre-combined variant would reverse just fine, but the renderer-combined á would reverse to ´a, which is probably not what was intended. If you're dealing with Unicode you need to be dealing with a good library and you need to be understanding what you're doing, once you go away from simply reading/displaying/serializing bags-o-bits. But if the latter is all you're doing, UTF-8 is just fine, and less susceptible to mistakes with code that works correctly on C strings.
cremno 12y ago
>it wasn't in Visual Studio last I checked The upcoming version has it.

syncsynchalt 12y ago

Unicode code points can only take 4 bytes in UTF-8, this has been true ever since the planes were capped at U+10FFFF. Or did you mean UTF-8 encoded surrogate pairs?

maxlybbert 12y ago
Thanks for the correction. I don't know where I originally got the "up top six bytes" number from, but I've been using it for a while. Apparently, it's out of date (looking at the original proposal, https://en.wikipedia.org/wiki/UTF-8#Description https://en.wikipedia.org/wiki/UTF-8#Description , some code points were expected to need 6 bytes, but as you say, that's no longer true).
- syncsynchalt 12y ago
  It was true until 2003, so in this case reality has been retconned on you.

nly 12y ago

UTF-16 has surrogate pairs as well. It's the worst of both worlds.

maxlybbert 12y ago
I prefer UTF-32, but I'm very lonely. The benefit with UTF-16 is that you can't accidentally pass a string to, say, strlen(). But, yes, people will forget that not all Unicode code points fit in 16 bits, and won't test with the right kind of input to find out that they've done it wrong. So errors can still creep in; but I will continue to argue that those errors are less common because the fact that you're dealing with Unicode (and not ASCII or something else) is more obvious and you're more likely to need library calls (that will do the right thing) to do anything useful.
- masklinn 12y ago
  > I prefer UTF-32, but I'm very lonely. I'm not sure why you'd prefer UTF-32, it doesn't make correct text manipulations any easier, but — much like UTF-16 — it does make incorrect assumptions and text manipulations much easier.
  - maxlybbert 12y ago
    UTF-32 means each code point takes up one char32_t. There are no surrogate pairs. However, it is true that you still have to deal with combining characters, normalization, etc. So it doesn't solve all of your problems, but it does avoid simple goofiness ( http://blog.coverity.com/2014/04/09/why-utf-16/ http://blog.coverity.com/2014/04/09/why-utf-16/ ). You still have to use libraries to do anything non-trivial.
    - masklinn 12y ago
      Right. So while UTF-16 is a terrible encoding and there is very little reason to use it, UTF-32 is little better. Use UTF-8.
      - maxlybbert 12y ago
        Like I said, I'm very lonely. My only consolation is that every project I've seen that says "use UTF-8 and be careful not to do goofy things (like running a string through the encoder twice, or passing a string to old C string functions that aren't Unicode aware, or assuming each character takes one byte), without exception, constantly finds those goofy mistakes in their code. Being careful is not enough. UTF-8 for strings on disk, yes. But UTF-8 for in-memory strings has a long track record of being much harder to actually do correctly.

mpyne 12y ago

Beyond the issue noted with UTF-16 permitting surrogate pairs, it's also important to understand that UTF-16 does not free you from normalization form concerns. For instance, even within the 16-bit Basic Multilingual Plane, it is not safe to reverse a UTF-16 string by simply reversing each block of 2 bytes, as a Unicode glyph (e.g. á) can be composed of a pre-combined code point which only takes 16-bits, or a base character (a) followed by a combiner (´). The pre-combined variant would reverse just fine, but the renderer-combined á would reverse to ´a, which is probably not what was intended. If you're dealing with Unicode you need to be dealing with a good library and you need to be understanding what you're doing, once you go away from simply reading/displaying/serializing bags-o-bits. But if the latter is all you're doing, UTF-8 is just fine, and less susceptible to mistakes with code that works correctly on C strings.

cremno 12y ago

>it wasn't in Visual Studio last I checked The upcoming version has it.