5 ms·

Nobody's ever accidentally tried to uppercase a UTF-32 string by passing each code point to C's toupper() function; but that constantly shows up in projects usi

Nobody's ever accidentally tried to uppercase a UTF-32 string by passing each code point to C's toupper() function; but that constantly shows up in projects using UTF-8 for in-memory strings. And it's not always caught by unit tests. People never accidentally pass a UTF-32 string to a legacy API that expects char*'s in Latin1 encoding. I when I say "never" I don't mean "almost never happens"; I mean literally "anybody who tries, gets a compile time error." With UTF-32, you don't have to worry about copying less than a full code point to a destination string, but if you forget that possibility with UTF-8 or UTF-16 you can actually create security problems. And no matter how much work people do to fix the code, those mistakes creep in repeatedly. Yes, people forget to normalize their UTF-32, UTF-16 and UTF-8 strings before comparing for equality. Yes, people forget that whether a code point is a letter depends on who's asking the question (I usually don't consider Greek letter pi a letter, but Greeks do). Yes, it's true that reversing a Unicode string in any encoding requires more than simply reversing the individual elements (because of combining characters). So, yes, it's possible to get things wrong in any encoding. But UTF-8 has more ways to screw up than the alternatives. And UTF-16 has a few more than UTF-32; but I can accept UTF-16 if there are external reasons to (e.g., working on Windows).

cygx 12y ago
> With UTF-32, you don't have to worry about copying less than a full code point to a destination string But you can still copy only parts of a grapheme cluster. If you want to do Unicode right, you have to treat even UTF-32 as a variable-length coding.
- maxlybbert 12y ago
  To do Unicode right, you have to know what you mean by "character," which usually means never saying "character" and instead being more precise ("code point," "grapheme cluster," "byte representation"). You also have to figure out what people expect to happen when they hit backspace (does the whole grapheme cluster disappear? the most-recently-typed code point in the cluster disappear? do you have to hit backspace twice to get rid of surrogate pairs (a la Notepad on Windows)? should you have to hit backspace a variable number of times to delete code points encoded in UTF-8 (following the Notepad behavior to its logical conclusion)?).
masklinn 12y ago
I will repeat, since you apparently managed to miss it last time around: using your type system does not require using UTF32 as your internal string encoding. Hell, you don't even need a type system to have separate bytes and string types, you can even do that in dynamically typed languages, regardless of the string type's internal encoding (it could even be variable, within a single string).
- maxlybbert 12y ago
  I didn't miss it. I simply know projects that use UTF-8 for in-memory strings, and none of the ones I'm familiar with uses a different type for it. It's all convention, and they all occasionally find that somebody flubbed the convention. Can you point me to any projects that manipulate UTF-8 encoded in-memory strings and actually use a different type for it them? I realize I won't convince you. That's what I meant in the original comment that I know many people disagree with me on this.
  - mpyne 12y ago
    > Can you point me to any projects that manipulate UTF-8 encoded in-memory strings and actually use a different type for it them? Glib seems to. https://developer.gnome.org/glib/2.37/glib-Strings.html https://developer.gnome.org/glib/2.37/glib-Strings.html Though in complete fairness the type appears to be just a "bags of bytes" type and so could hold anything, it's really meant to hold UTF-8 (as you can tell from the functions that append/prepend Unicode chars).
    - maxlybbert 12y ago
      Thanks for the pointer.
  - masklinn 12y ago
    > I simply know projects that use UTF-8 for in-memory strings, and none of the ones I'm familiar with uses a different type for it. Which is relevant… how? > Can you point me to any projects that manipulate UTF-8 encoded in-memory strings and actually use a different type for it them? Rust does. Python 3.3 does something similar but slightly different (it switches internal representation between iso-8859-1, UCS2 and UCS4 depending on the string's codepoints). > I realize I won't convince you. That's what I meant in the original comment that I know many people disagree with me on this. Of course you won't convince me, your original comment is based on inane premises.
nly 12y ago
> Nobody's ever accidentally tried to uppercase a UTF-32 string by passing each code point to C's toupper() function Note to put aside, since this was all about C++, the toupper() in <locale> is a template type that takes an arbitrary character type and a locale. There are also functions for locale sensitive collation and comparison. So, modulo bad implementation, you should be able to do basic unicode string handling in ISO C++. The GNU stdlibc++ manual goes in to quite a lot of detail: https://gcc.gnu.org/onlinedocs/libstdc++/manual/localization.html https://gcc.gnu.org/onlinedocs/libstdc++/manual/localization...
- maxlybbert 12y ago
  > So, modulo bad implementation, you should be able to do basic unicode string handling in ISO C++. The locale stuff has been intentionally left vague. So: (1) assuming that your implementation supports passing a char16_t or char32_t to the functions in <locale>, you're guaranteed it will do the right thing as far as Unicode and the C++ standard (although, there are some well-known problems with what the C++ standard defines the "right thing" top be for lowercasing epsilon -- since in Greek, there are two lowercase forms of epsilon, and the C++ standard doesn't provide the function enough information to decide between those two forms -- and for uppercasing ẞ (LATIN CAPITAL LETTER SHARP S) -- because in German the uppercase form takes two letters, and the C++ standard assumes doesn't allow for that). The fun part is that plain char's can be encoded in several different ways, and not all have anything to do with Unicode. So you need to have a Unicode-aware locale for the functions in <locale> to do anything sensible with Unicode. Of course, that's something of a tautology, but I'm not sure what the standard requires for locales. So, yes, if your implementation is Unicode aware, you can call a standard C++ function to get what you want, but be sure to call the right one. There's another, with the same name, that isn't guaranteed to do what you want.

cygx 12y ago

> With UTF-32, you don't have to worry about copying less than a full code point to a destination string But you can still copy only parts of a grapheme cluster. If you want to do Unicode right, you have to treat even UTF-32 as a variable-length coding.

maxlybbert 12y ago
To do Unicode right, you have to know what you mean by "character," which usually means never saying "character" and instead being more precise ("code point," "grapheme cluster," "byte representation"). You also have to figure out what people expect to happen when they hit backspace (does the whole grapheme cluster disappear? the most-recently-typed code point in the cluster disappear? do you have to hit backspace twice to get rid of surrogate pairs (a la Notepad on Windows)? should you have to hit backspace a variable number of times to delete code points encoded in UTF-8 (following the Notepad behavior to its logical conclusion)?).

masklinn 12y ago

I will repeat, since you apparently managed to miss it last time around: using your type system does not require using UTF32 as your internal string encoding. Hell, you don't even need a type system to have separate bytes and string types, you can even do that in dynamically typed languages, regardless of the string type's internal encoding (it could even be variable, within a single string).

maxlybbert 12y ago
I didn't miss it. I simply know projects that use UTF-8 for in-memory strings, and none of the ones I'm familiar with uses a different type for it. It's all convention, and they all occasionally find that somebody flubbed the convention. Can you point me to any projects that manipulate UTF-8 encoded in-memory strings and actually use a different type for it them? I realize I won't convince you. That's what I meant in the original comment that I know many people disagree with me on this.
- mpyne 12y ago
  > Can you point me to any projects that manipulate UTF-8 encoded in-memory strings and actually use a different type for it them? Glib seems to. https://developer.gnome.org/glib/2.37/glib-Strings.html https://developer.gnome.org/glib/2.37/glib-Strings.html Though in complete fairness the type appears to be just a "bags of bytes" type and so could hold anything, it's really meant to hold UTF-8 (as you can tell from the functions that append/prepend Unicode chars).
  - maxlybbert 12y ago
    Thanks for the pointer.
- masklinn 12y ago
  > I simply know projects that use UTF-8 for in-memory strings, and none of the ones I'm familiar with uses a different type for it. Which is relevant… how? > Can you point me to any projects that manipulate UTF-8 encoded in-memory strings and actually use a different type for it them? Rust does. Python 3.3 does something similar but slightly different (it switches internal representation between iso-8859-1, UCS2 and UCS4 depending on the string's codepoints). > I realize I won't convince you. That's what I meant in the original comment that I know many people disagree with me on this. Of course you won't convince me, your original comment is based on inane premises.

nly 12y ago

> Nobody's ever accidentally tried to uppercase a UTF-32 string by passing each code point to C's toupper() function Note to put aside, since this was all about C++, the toupper() in <locale> is a template type that takes an arbitrary character type and a locale. There are also functions for locale sensitive collation and comparison. So, modulo bad implementation, you should be able to do basic unicode string handling in ISO C++. The GNU stdlibc++ manual goes in to quite a lot of detail: https://gcc.gnu.org/onlinedocs/libstdc++/manual/localization.html https://gcc.gnu.org/onlinedocs/libstdc++/manual/localization...

maxlybbert 12y ago
> So, modulo bad implementation, you should be able to do basic unicode string handling in ISO C++. The locale stuff has been intentionally left vague. So: (1) assuming that your implementation supports passing a char16_t or char32_t to the functions in <locale>, you're guaranteed it will do the right thing as far as Unicode and the C++ standard (although, there are some well-known problems with what the C++ standard defines the "right thing" top be for lowercasing epsilon -- since in Greek, there are two lowercase forms of epsilon, and the C++ standard doesn't provide the function enough information to decide between those two forms -- and for uppercasing ẞ (LATIN CAPITAL LETTER SHARP S) -- because in German the uppercase form takes two letters, and the C++ standard assumes doesn't allow for that). The fun part is that plain char's can be encoded in several different ways, and not all have anything to do with Unicode. So you need to have a Unicode-aware locale for the functions in <locale> to do anything sensible with Unicode. Of course, that's something of a tautology, but I'm not sure what the standard requires for locales. So, yes, if your implementation is Unicode aware, you can call a standard C++ function to get what you want, but be sure to call the right one. There's another, with the same name, that isn't guaranteed to do what you want.