6 ms·

I prefer UTF-32, but I'm very lonely. The benefit with UTF-16 is that you can't accidentally pass a string to, say, strlen(). But, yes, people will forget that

I prefer UTF-32, but I'm very lonely. The benefit with UTF-16 is that you can't accidentally pass a string to, say, strlen(). But, yes, people will forget that not all Unicode code points fit in 16 bits, and won't test with the right kind of input to find out that they've done it wrong. So errors can still creep in; but I will continue to argue that those errors are less common because the fact that you're dealing with Unicode (and not ASCII or something else) is more obvious and you're more likely to need library calls (that will do the right thing) to do anything useful.

masklinn 12y ago
> I prefer UTF-32, but I'm very lonely. I'm not sure why you'd prefer UTF-32, it doesn't make correct text manipulations any easier, but — much like UTF-16 — it does make incorrect assumptions and text manipulations much easier.
- maxlybbert 12y ago
  UTF-32 means each code point takes up one char32_t. There are no surrogate pairs. However, it is true that you still have to deal with combining characters, normalization, etc. So it doesn't solve all of your problems, but it does avoid simple goofiness ( http://blog.coverity.com/2014/04/09/why-utf-16/ http://blog.coverity.com/2014/04/09/why-utf-16/ ). You still have to use libraries to do anything non-trivial.
  - masklinn 12y ago
    Right. So while UTF-16 is a terrible encoding and there is very little reason to use it, UTF-32 is little better. Use UTF-8.
    - maxlybbert 12y ago
      Like I said, I'm very lonely. My only consolation is that every project I've seen that says "use UTF-8 and be careful not to do goofy things (like running a string through the encoder twice, or passing a string to old C string functions that aren't Unicode aware, or assuming each character takes one byte), without exception, constantly finds those goofy mistakes in their code. Being careful is not enough. UTF-8 for strings on disk, yes. But UTF-8 for in-memory strings has a long track record of being much harder to actually do correctly.
      - TheLoneWolfling 12y ago
        What I want: A language where the "String" data type is as follows: A rope of "logical characters" (One or more code points, such that they are logically one character. So an accent is combined with the previous character, that sort of thing.) With the additional "restriction" (read: implementation detail) that within a single node all logical characters must have the same width. (You can, for example, store a single one-byte character in a run of two-byte characters as an overlong-encoded two-byte character, but this is just an optimization.) Short ropes degenerate to a flat array. (You have to do a workaround for single code points that encode multiple logical characters. You split them into N parts encoded in the private unicode range or something similar, and when displaying them recombine them if they are in the correct order, otherwise normalize them. Although I'm up in the air about this. Should reverse("ﬆ") be "ﬆ"? Or "ts"? (That's the single unicode character "ﬆ", for those that are confused.)) Ideally, you put character encoding directly within nodes. That way most things "just work". Running a string through the encoder twice doesn't do anything, as it detects the encoding is the same as the target encoding and doesn't do anything. Reversing a string "just works". Indexing a string is sub-linear time, but gives decent results. (Indexing a string and getting invalid unicode as a result is never fun!) Concatenating strings takes sublinear time even. This works really well with immutable data structures, or quasi-immutable data structures. (there's some tricks with rewriting ropes to take maximal advantage of structure-sharing that preserve the illusion of an immutable data structure without actually being immutable.) And if you really want you can start doing fancy things like allowing lazy generators within strings, or lazily decompressing / reading data from disk. To store on disk? Yeah, go with UTF-8. (Or my personal favorite pet encoding: compressed UTF-32.)
      - masklinn 12y ago
        > Being careful is not enough. UTF32 will not help you much with that, save that you've got a separate type for "strings" and "bunch of bytes" Which you can have anyway, so do that, it's a good idea which doesn't require using UTF32. > But UTF-8 for in-memory strings has a long track record of being much harder to actually do correctly. As if other in-memory encodings had a better track record.
        12y ago

masklinn 12y ago

> I prefer UTF-32, but I'm very lonely. I'm not sure why you'd prefer UTF-32, it doesn't make correct text manipulations any easier, but — much like UTF-16 — it does make incorrect assumptions and text manipulations much easier.

maxlybbert 12y ago
UTF-32 means each code point takes up one char32_t. There are no surrogate pairs. However, it is true that you still have to deal with combining characters, normalization, etc. So it doesn't solve all of your problems, but it does avoid simple goofiness ( http://blog.coverity.com/2014/04/09/why-utf-16/ http://blog.coverity.com/2014/04/09/why-utf-16/ ). You still have to use libraries to do anything non-trivial.
- masklinn 12y ago
  Right. So while UTF-16 is a terrible encoding and there is very little reason to use it, UTF-32 is little better. Use UTF-8.
  - maxlybbert 12y ago
    Like I said, I'm very lonely. My only consolation is that every project I've seen that says "use UTF-8 and be careful not to do goofy things (like running a string through the encoder twice, or passing a string to old C string functions that aren't Unicode aware, or assuming each character takes one byte), without exception, constantly finds those goofy mistakes in their code. Being careful is not enough. UTF-8 for strings on disk, yes. But UTF-8 for in-memory strings has a long track record of being much harder to actually do correctly.
    - TheLoneWolfling 12y ago
      What I want: A language where the "String" data type is as follows: A rope of "logical characters" (One or more code points, such that they are logically one character. So an accent is combined with the previous character, that sort of thing.) With the additional "restriction" (read: implementation detail) that within a single node all logical characters must have the same width. (You can, for example, store a single one-byte character in a run of two-byte characters as an overlong-encoded two-byte character, but this is just an optimization.) Short ropes degenerate to a flat array. (You have to do a workaround for single code points that encode multiple logical characters. You split them into N parts encoded in the private unicode range or something similar, and when displaying them recombine them if they are in the correct order, otherwise normalize them. Although I'm up in the air about this. Should reverse("ﬆ") be "ﬆ"? Or "ts"? (That's the single unicode character "ﬆ", for those that are confused.)) Ideally, you put character encoding directly within nodes. That way most things "just work". Running a string through the encoder twice doesn't do anything, as it detects the encoding is the same as the target encoding and doesn't do anything. Reversing a string "just works". Indexing a string is sub-linear time, but gives decent results. (Indexing a string and getting invalid unicode as a result is never fun!) Concatenating strings takes sublinear time even. This works really well with immutable data structures, or quasi-immutable data structures. (there's some tricks with rewriting ropes to take maximal advantage of structure-sharing that preserve the illusion of an immutable data structure without actually being immutable.) And if you really want you can start doing fancy things like allowing lazy generators within strings, or lazily decompressing / reading data from disk. To store on disk? Yeah, go with UTF-8. (Or my personal favorite pet encoding: compressed UTF-32.)
    - masklinn 12y ago
      > Being careful is not enough. UTF32 will not help you much with that, save that you've got a separate type for "strings" and "bunch of bytes" Which you can have anyway, so do that, it's a good idea which doesn't require using UTF32. > But UTF-8 for in-memory strings has a long track record of being much harder to actually do correctly. As if other in-memory encodings had a better track record.
      - 12y ago