4 ms·

Like I said, I'm very lonely. My only consolation is that every project I've seen that says "use UTF-8 and be careful not to do goofy things (like running a str

Like I said, I'm very lonely. My only consolation is that every project I've seen that says "use UTF-8 and be careful not to do goofy things (like running a string through the encoder twice, or passing a string to old C string functions that aren't Unicode aware, or assuming each character takes one byte), without exception, constantly finds those goofy mistakes in their code. Being careful is not enough. UTF-8 for strings on disk, yes. But UTF-8 for in-memory strings has a long track record of being much harder to actually do correctly.

TheLoneWolfling 12y ago
What I want: A language where the "String" data type is as follows: A rope of "logical characters" (One or more code points, such that they are logically one character. So an accent is combined with the previous character, that sort of thing.) With the additional "restriction" (read: implementation detail) that within a single node all logical characters must have the same width. (You can, for example, store a single one-byte character in a run of two-byte characters as an overlong-encoded two-byte character, but this is just an optimization.) Short ropes degenerate to a flat array. (You have to do a workaround for single code points that encode multiple logical characters. You split them into N parts encoded in the private unicode range or something similar, and when displaying them recombine them if they are in the correct order, otherwise normalize them. Although I'm up in the air about this. Should reverse("ﬆ") be "ﬆ"? Or "ts"? (That's the single unicode character "ﬆ", for those that are confused.)) Ideally, you put character encoding directly within nodes. That way most things "just work". Running a string through the encoder twice doesn't do anything, as it detects the encoding is the same as the target encoding and doesn't do anything. Reversing a string "just works". Indexing a string is sub-linear time, but gives decent results. (Indexing a string and getting invalid unicode as a result is never fun!) Concatenating strings takes sublinear time even. This works really well with immutable data structures, or quasi-immutable data structures. (there's some tricks with rewriting ropes to take maximal advantage of structure-sharing that preserve the illusion of an immutable data structure without actually being immutable.) And if you really want you can start doing fancy things like allowing lazy generators within strings, or lazily decompressing / reading data from disk. To store on disk? Yeah, go with UTF-8. (Or my personal favorite pet encoding: compressed UTF-32.)
masklinn 12y ago
> Being careful is not enough. UTF32 will not help you much with that, save that you've got a separate type for "strings" and "bunch of bytes" Which you can have anyway, so do that, it's a good idea which doesn't require using UTF32. > But UTF-8 for in-memory strings has a long track record of being much harder to actually do correctly. As if other in-memory encodings had a better track record.
- maxlybbert 12y ago
  Nobody's ever accidentally tried to uppercase a UTF-32 string by passing each code point to C's toupper() function; but that constantly shows up in projects using UTF-8 for in-memory strings. And it's not always caught by unit tests. People never accidentally pass a UTF-32 string to a legacy API that expects char*'s in Latin1 encoding. I when I say "never" I don't mean "almost never happens"; I mean literally "anybody who tries, gets a compile time error." With UTF-32, you don't have to worry about copying less than a full code point to a destination string, but if you forget that possibility with UTF-8 or UTF-16 you can actually create security problems. And no matter how much work people do to fix the code, those mistakes creep in repeatedly. Yes, people forget to normalize their UTF-32, UTF-16 and UTF-8 strings before comparing for equality. Yes, people forget that whether a code point is a letter depends on who's asking the question (I usually don't consider Greek letter pi a letter, but Greeks do). Yes, it's true that reversing a Unicode string in any encoding requires more than simply reversing the individual elements (because of combining characters). So, yes, it's possible to get things wrong in any encoding. But UTF-8 has more ways to screw up than the alternatives. And UTF-16 has a few more than UTF-32; but I can accept UTF-16 if there are external reasons to (e.g., working on Windows).
  - cygx 12y ago
    > With UTF-32, you don't have to worry about copying less than a full code point to a destination string But you can still copy only parts of a grapheme cluster. If you want to do Unicode right, you have to treat even UTF-32 as a variable-length coding.
    - 12y ago

TheLoneWolfling 12y ago

What I want: A language where the "String" data type is as follows: A rope of "logical characters" (One or more code points, such that they are logically one character. So an accent is combined with the previous character, that sort of thing.) With the additional "restriction" (read: implementation detail) that within a single node all logical characters must have the same width. (You can, for example, store a single one-byte character in a run of two-byte characters as an overlong-encoded two-byte character, but this is just an optimization.) Short ropes degenerate to a flat array. (You have to do a workaround for single code points that encode multiple logical characters. You split them into N parts encoded in the private unicode range or something similar, and when displaying them recombine them if they are in the correct order, otherwise normalize them. Although I'm up in the air about this. Should reverse("ﬆ") be "ﬆ"? Or "ts"? (That's the single unicode character "ﬆ", for those that are confused.)) Ideally, you put character encoding directly within nodes. That way most things "just work". Running a string through the encoder twice doesn't do anything, as it detects the encoding is the same as the target encoding and doesn't do anything. Reversing a string "just works". Indexing a string is sub-linear time, but gives decent results. (Indexing a string and getting invalid unicode as a result is never fun!) Concatenating strings takes sublinear time even. This works really well with immutable data structures, or quasi-immutable data structures. (there's some tricks with rewriting ropes to take maximal advantage of structure-sharing that preserve the illusion of an immutable data structure without actually being immutable.) And if you really want you can start doing fancy things like allowing lazy generators within strings, or lazily decompressing / reading data from disk. To store on disk? Yeah, go with UTF-8. (Or my personal favorite pet encoding: compressed UTF-32.)

masklinn 12y ago

> Being careful is not enough. UTF32 will not help you much with that, save that you've got a separate type for "strings" and "bunch of bytes" Which you can have anyway, so do that, it's a good idea which doesn't require using UTF32. > But UTF-8 for in-memory strings has a long track record of being much harder to actually do correctly. As if other in-memory encodings had a better track record.

maxlybbert 12y ago
Nobody's ever accidentally tried to uppercase a UTF-32 string by passing each code point to C's toupper() function; but that constantly shows up in projects using UTF-8 for in-memory strings. And it's not always caught by unit tests. People never accidentally pass a UTF-32 string to a legacy API that expects char*'s in Latin1 encoding. I when I say "never" I don't mean "almost never happens"; I mean literally "anybody who tries, gets a compile time error." With UTF-32, you don't have to worry about copying less than a full code point to a destination string, but if you forget that possibility with UTF-8 or UTF-16 you can actually create security problems. And no matter how much work people do to fix the code, those mistakes creep in repeatedly. Yes, people forget to normalize their UTF-32, UTF-16 and UTF-8 strings before comparing for equality. Yes, people forget that whether a code point is a letter depends on who's asking the question (I usually don't consider Greek letter pi a letter, but Greeks do). Yes, it's true that reversing a Unicode string in any encoding requires more than simply reversing the individual elements (because of combining characters). So, yes, it's possible to get things wrong in any encoding. But UTF-8 has more ways to screw up than the alternatives. And UTF-16 has a few more than UTF-32; but I can accept UTF-16 if there are external reasons to (e.g., working on Windows).
- cygx 12y ago
  > With UTF-32, you don't have to worry about copying less than a full code point to a destination string But you can still copy only parts of a grapheme cluster. If you want to do Unicode right, you have to treat even UTF-32 as a variable-length coding.
  - 12y ago