7 ms·
Should UTF-16 be considered harmful?
- xpaulbettsx 15y agoUTF-16 has all of the disadvantages of UTF-8 and none of the advantages, and originally comes from the UCS-2 era where they thought, "64k characters are enough for everyone!" Unfortunately, all of Windows uses it, so we as an industry are stuck with it.
- jensnockert 15y agoYou mean that it is not ASCII compatible? Also, Mac OS X, Java, .Net, Qt, ICU... there are a lot of support for UTF-16, for other reasons than backwards compatibility. Processing UTF-16 is easier in many situations.
- nitrogen 15y agoProcessing UTF-16 would only be easier if you have a valid byte-order mark and/or know the endianness in advance and you can guarantee that no surrogate pairs exist. Otherwise the same pitfalls exist as when processing UTF-8, plus the endianness issue and the possibility of UTF-16 programmers not knowing about surrogate pairs.
- xpaulbettsx 15y agoWhat you're describing is UCS-2, not UTF-16. That's why the latter is frustrating to deal with
- sp332 15y agoUTF-16 is not ASCII-compatible, if that's what you were asking.
- jensnockert 15y agoUnicode should be considered harmful, possibly even text. Never think you understand text, it is a very complex medium, and every time this topic is brought up, you learn something about some odd quality of some language that you might never have heard of. Yes, UTF-16 is variable length, yes, it does make many European scripts larger. Size is always a trade-off and there won't be one standard for encoding. Text is hard, do not approach it with a C library you built in an afternoon, leave it to the professionals. I just wish I knew any...
- yuhong 15y agoICU is one library.
- pyre 15y agotchrist seems pretty knowledgeable on unicode issues. He even did 3 talks at OSCON on the topic. My takeaway wasthat the best language for dealing with Unicode was Perl, and Ruby was the second best.
- compay 15y agoI gave a presentation on this topic at Rubyconf Brazil last year, and would be hard pressed to describe Ruby as "good" at dealing with Unicode, unless by "good" you mean "avoids making almost any decisions at all" (which might actually be a good thing but it's debatable). Ruby 1.9 doesn't even offer Unicode case folding, so from a practical standpoint working with Unicode text is a PITA with Ruby unless you use third party libraries. Ruby source can include constants, variables, etc. with Unicode (or other character set) symbols, which is very cool but for Unicode text processing I've found Ruby to be frustratingly lacking.
- pyre 15y agoHere are the slides for his "Unicode Shootout: The Good, The Bad, and the Ugly": http://code.activestate.com/lists/perl5-porters/166738/ http://code.activestate.com/lists/perl5-porters/166738/ Hmmm... According to that email he says that Java is second, but in his first talk he did say that Ruby (1.9+) was second in his mind (and he seemed to be visibily frustrated with Java's Unicode support).
- deleted 15y ago[deleted]
- derefr 15y ago> Size is always a trade-off and there won't be one standard for encoding. I don't see why, in modern multi-core CPUs/GPUs where 99% of time is spent idle, we can't just have "raw" text (UCS4) to manipulate in memory, and compressed text (using any standard stream compression algorithm) on disk/in the DB/over the wire. Anything that's not UCS4 is already variable-length-encoded, so you lose the ability to random-seek it anyway; and (safely performing) any complex text manipulation, e.g. upper-casing, requires temporarily converting the text to UCS4 anyway. At that point, you may as well go all the way, and serialize it as efficiently as possible, if you're just going to spit it out somewhere else. I guess the only difference is that string-append operations would require un-compressing compressed strings and then re-compressing the result—but you could defer that as long as necessary using rope[1]. [1] http://en.wikipedia.org/wiki/Rope_(computer_science) http://en.wikipedia.org/wiki/Rope_(computer_science)
- kogir 15y agoIf whatever framework or libraries you're using don't hide this from you nearly completely, you might be doing it wrong. Just use the default Unicode format of the OS and framework or standard libraries you're using and never think of it again. Much like dates and times, correct handling of Unicode is best left to the poor souls stuck dealing with them full time, all the time.
- nyellin 15y agoThat is not an option for low-level programming.
- maaku 15y ago"correct handling of Unicode is best left to the poor souls stuck dealing with them full time, all the time"
- hrktb 15y ago> Much like dates and times, correct handling of Unicode is best left to the poor souls stuck dealing with them full time, all the time. Reminds me that date parsing is fucked up in iOs and you actually need to know your shit to get it right when dealing with users around the globe. But I don't see people dealing with this full time.
- chubs 15y agoIn all apps i've worked on, passing dates around as 'seconds since 1970' has made date parsing on ios effortless - you may want to try that.
- klodolph 15y agoGive me a call in 2038.
- pyre 15y ago> passing dates around as 'seconds since 1970' has > made date parsing on ios effortless I'm confused. How does the way that you store dates internally have anything to do with date parsing? If you are accepting dates from users, then you still have to normalize then to unix timestamps. That also doesn't cover things like: * Take this date and convert it into a week represented by the last Sunday of the week (i.e. 20110410 represents 20110403-20110409). * Basically anything involving days of the week.
- cpeterso 15y agoIs there any reason to use a character encoding other than UTF-8 (for new content)? There is a (lot of) legacy content in non-Unicode encodings. Also, UCS-4 might be a useful optimization for processing text.
- maaku 15y agoSize, perhaps? Remember, most of the world does not speak English. I bet you baidu uses UTF-16 for its indexes.
- burgerbrain 15y agoIs size actually a concern in the modern world? With transmission and storage size is practically a nonissue and we have proper rather cheap compression anyway. And in memory? We have gigabytes of ram, who cares? Of the potential situations where it may become an issue that I can think of, pretty much none of them are on end user hardware.
- maaku 15y agoI reiterate my example of baidu. Just because size isn't a problem for your application, doesn't mean there aren't others out there dealing with petabytes of text.
- burgerbrain 15y ago"Of the potential situations where it may become an issue that I can think of, pretty much none of them are on end user hardware." Do what you need to on your companies machines, but keep it out of ecosystems it has no reason to be in.
- hrktb 15y agothe other way round, having a wildly variable size depending on the character can be a weakness. In japanese EUC was sometime used despite its shortcomings to force 16bit by character.
- 15y ago
- jrockway 15y agoHarmful, not really. Sometimes UTF-16 is the most compact representation for a given string, and its semantics are not that confusing. UTF-8 is a better "default choice", but the algorithms for both are very hard to get right. Given an idealized Unicode string object, it's still hard to do things like count the number of glyphs required to display the text in font foo, or convert all characters to "lowercase", or sort a list of names in "alphebetical" order. But the problem is not Unicode or UTF-16 or UTF-8, the problem is defining what a correct computer program is, and ultimately, that's the job for the programmer, not the string encoding algorithm. Think of UTF-8 and UTF-16 as being like in-memory compression for Unicode strings. You can't take a gzip file, count the number of bytes, and multiply by some factor to get the true length of the string. But, this complexity is often worth the space savings, so that's why these encodings exist. The problem that people have is twofold: one, the average programmer doesn't realize that characters and character encodings are two different things. The second is, they don't realize that a Unicode string is a data structure that does one thing and only one thing: maintain a sequence of Unicode codepoints in memory. A Unicode string is not "text" in some language; it's some characters for a computer. To choose the right font or sort things correctly, you need additional information sent out-of-band. There is no way to do it right given only the Unicode string itself. People are reluctant to send out-of-bad data and so other people assume Unicode is hard. It's not. The problem is that Unicode is not text.
- maaku 15y agoYou shouldn't care about UTF-8/UTF-16/UCS-4 except for performance, and that totally depends on what you are doing with what data-sets. As someone who speaks multiple languages and has written a fair amount of language-processing code, the simple truth is that if you are not using a vetted unicode framework for writing your application--and if you're asking if UTF-16 should be considered harmful, you're probably not--you are almost certainly introducing massive numbers of bugs that your own cultural biases are blinding you to. Do not underestimate the difficulty of writing correct multi-lingual-aware programs. These frameworks exist for a reason, and are often written by professionally trained linguists.
- archangel_one 15y agoInterfacing with third-party libraries is an obvious case where you do need to worry about encodings for reasons that are not performance.
- PaulHoule 15y agoMy experience is that the harder a framework tries to "support" Unicode correctly, the more likely it's going to have bugs ranging from annoying to showstopper. Simply passing UTF-8 through almost works if it wasn't for the fact that "pass through" allows illegal character combinations that can cause all sorts of trouble.
- thisrod 15y agoI see a different problem: the scope of Unicode has crept outside the scope of its designers' expertise. Next you know, we'll have a code point for the symbol "arctanh". (Moreover, sin through arccosh will be contiguous, erf will be placed where you'd expect arctanh, and arctanh will sit between blackboard bold B and D, because C is listed a trillion code points earlier as "Variant Complex Number Sign", between "Engineering Right Angle Bracket" and "Computer Programming Left Parenthesis".) Encoding the technical and mathematical symbols in two bytes is necessarily kludgy. There are International Standard kludges that everyone is supposed to use, but not everyone does. My solution would be for Unicode to stick to human languages.
- caf 15y agoI agree with much of what the top-voted answer says, with the exception of the criticism of wchar_t. wchar_t is not equal to utf-16: it is supposed to be a type large enough to store any single codepoint. Environments where wchar_t is utf-16 simply have a broken wchar_t - when wchar_t is ucs4 then it fulfills its original purpose.
- pilif 15y agoUTF16 gives you the worst of both worlds: it wastes a lot of memory for ASCII-like languages (mostly ASCII plus some special characters), you have to deal with byte ordering and you still don't get the advantages of directly addressing a specific character without parsing the whole string up to the character you want. But. If you widen the character even more, you'd probably still want it to be somewhat word-aligned so you'd use 32bit per character which would have enough storage to store all of Unicode plus some. The cost though is obvious: you waste memory. Depending on the language, you even waste a lot of memory: think ASCII or ASCII-like (the middle european languages). In UHF-8 those need - depending on language - barely more than one byte up to two bytes per character. Representing these languages with 4 bytes per character makes you use nearly 4 times the memory reasonably needed. This changes the farther east you move. Representing Chinese (no ASCII, many character points high up in the set) in utf-8, you begin wasting a lot of memory due to the ASCII compatibility. As encoding a Unicode code point in utf-8 uses around one byte more than if you would just store the code point as an integer. So on international software running on potentially limited memory while targeting the eastern languages, you will again be better off using utf-16 as it requires less storage for characters really high up in the Unicode plane. Also, if you know that you are just extended ASCII, you can optimize and access characters directly without parsing the whole string, giving you another speed advantage. I don't know what's the best way to go. 32 bits is wasteful, utf-16 is sometimes wasteful, has endianness issues and still needs parsing (but is less wasteful than 32bits in most realistic cases) and utf-8 is really wasteful for high code points and always requires parsing but doesn't have the endianness issues. I guess as always these are just tools and you have to pick what works in your situation. Developers have to adapt to what was picked.
- rwmj 15y agoBut as one of the replies in that thread pointed out, if you want to compress strings, use LZ! UTF-16 won't give you as good compression.
- EdiX 15y ago> If you widen the character even more (...) If you go to UTF32 you waste a lot more space in any conceivable situation (extra-BMP characters are less tha 0.1% of the text even in CJK languages) and you still don't get anything in exchange thanks to combining diacriticals and ligatures. Thanks Unicode Committee!
- j_baker 15y agoI dislike UTF16 for a very simple reason: it's another type of encoding to deal with. If I could force the entire world to dump UTF16 and use UTF8, I would. Hell, I'd force everyone to dump UTF8 and use UTF16 if I could too. It's just such a pain in the ass to have to deal with having so many ways of encoding text on the web.
- perlgeek 15y agoUTF-16 is not harmful, just the "I use UTF-16, so I don't have to worry about Unicode issues anymore" stance is harmful. Processing text is hard. For example if you write an editor, you must be aware of things like right-to-left text sequences, full-width characters and graphemes composed of multiple codepoints. If you implement a case insensitive search, you must be aware of title case, and the nasty German ß which case-folds to two upper-case characters (SS). (Ok, you'll probably just use regexes, it's the author of the regex engine who has to worry about it. And it is ugly, because it means that character classes can match two characters, which you have to consider in all length-based optimizations). In first approximation it doesn't matter if you implement your stuff in UTF-8 or in one of the UTF-16 encodings, you have to deal with the same issues (include variable byte length for a single codepoint).