3 ms·

The premise of the article wasn't that you shouldn't use Unicode. It was that you need to care about more than just Unicode. Thinking that the answer to localis

by lucumo 17y ago

The premise of the article wasn't that you shouldn't use Unicode. It was that you need to care about more than just Unicode. Thinking that the answer to localisation is to "just use Unicode" is a flaw in people's thinking that this article points out and corrects. I'm curious as to why you believe UTF-8 is the only encoding you should use? I know of at least one person that says you should use UTF-16, because then it's immediately obvious when a certain piece of data is in the wrong encoding, whereas with UTF-8 you need to look for non-Latin characters (or accents) to distinguish it from, for example, ISO-8859-1. That line of thinking holds some value to me. Why do you believe otherwise?

ajross 17y ago
UTF-16 can't be used with traditional tools. No grepping, no strings, no scanning quickly with a text editor. It locks you into whatever oddball toolchain you cooked up in development. Old code that passes the string to a strcat() will fail in mysterious ways. And your old libraries probably expect 1-byte characters anyway, which means you're going to be constantly converting The inability to distinguish between UTF-8 and ASCII is a feature, not a bug. If you want to know if it's a valid string, just validate it. UTF-8 has extraordinarily strong validation properties, it's almost impossible for strings in other encodings to parse successfully.
- lucumo 17y ago
  > Old code that passes the string to a strcat() will fail in mysterious ways. And your old libraries probably expect 1-byte characters anyway, which means you're going to be constantly converting But they will have that same problem with UTF-8, right? Just as soon as you start using non-Latin characters? And if you aren't, there's no point in going beyond ASCII.
  - ajross 17y ago
    Uh, no, not at all. The ANSI strXXX() functions work just fine with UTF-8, that was kind of the point. Anything treating strings with nul termination or counted byte length works without change. Any work involving only ASCII substrings (e.g. parsing of computer-readable data) works without change. No non-ASCII character is represented using ASCII bytes in the string. Everything Just Works. Which is why you should just use it. The sole exception are algorithms that truly need random access to characters within the string (e.g. "give me the 32nd character"), which can't interoperate with any variable-length encoding. But there are precious few of these. All anyone cares about is substrings.

by lucumo 17y ago

ajross 17y ago

UTF-16 can't be used with traditional tools. No grepping, no strings, no scanning quickly with a text editor. It locks you into whatever oddball toolchain you cooked up in development. Old code that passes the string to a strcat() will fail in mysterious ways. And your old libraries probably expect 1-byte characters anyway, which means you're going to be constantly converting The inability to distinguish between UTF-8 and ASCII is a feature, not a bug. If you want to know if it's a valid string, just validate it. UTF-8 has extraordinarily strong validation properties, it's almost impossible for strings in other encodings to parse successfully.

lucumo 17y ago
> Old code that passes the string to a strcat() will fail in mysterious ways. And your old libraries probably expect 1-byte characters anyway, which means you're going to be constantly converting But they will have that same problem with UTF-8, right? Just as soon as you start using non-Latin characters? And if you aren't, there's no point in going beyond ASCII.
- ajross 17y ago
  Uh, no, not at all. The ANSI strXXX() functions work just fine with UTF-8, that was kind of the point. Anything treating strings with nul termination or counted byte length works without change. Any work involving only ASCII substrings (e.g. parsing of computer-readable data) works without change. No non-ASCII character is represented using ASCII bytes in the string. Everything Just Works. Which is why you should just use it. The sole exception are algorithms that truly need random access to characters within the string (e.g. "give me the 32nd character"), which can't interoperate with any variable-length encoding. But there are precious few of these. All anyone cares about is substrings.