4 ms·
If you're representing text which contains lots of characters that aren't in ASCII, like say Chinese, UTF-8 will consume much more storage than necessary. There
by jharsman 16y ago
If you're representing text which contains lots of characters that aren't in ASCII, like say Chinese, UTF-8 will consume much more storage than necessary. There are many languages where non-ASCII characters are extremely common.
He misses a very useful property of UTF-8 as well, it never contains null bytes. This trips up all sorts of heuristics for detecting binary files in various programs if you use e.g. UTF-16 for mostly ASCII text, since it then will contain lots of nulls.
UTF-8 is a very clever way to avoid problems on systems suffering under the mistaken assumption that text is 8-bit byte strings (cough UNIX cough), but that doesn't make it the ideal choice every time.
It is still very common for cross platform tools to not handle file names with non-ASCII characters for example. Both Mercurial and Git suffered from this last I looked.
The reason is that they treat file names as byte strings instead of text in some encoding, and therefore cannot translate to the proper encoding on platforms which treat file names as Unicode text, like Windows and OS X. OS X also uses a somewhat unconventional normalization form, which means you need to handle normalization as well.
- pornel 16y ago> […] characters that aren't in ASCII, like say Chinese, UTF-8 will consume much more storage than necessary. I don't think that's a problem in practice. Taking for example: http://www.baidu.com/s?wd=%D0%C2+%CE%C5 http://www.baidu.com/s?wd=%D0%C2+%CE%C5 The page takes 37KB in GBK, 39KB in UTF-8 and 73KB in UTF-16! (UTF-16 doubles cost of HTML markup but only saves ⅓ in text) Even in pure Chinese plain text UTF-16 doesn't win by much. I've tested some random article: 18KB in UTF-16, 25KB in UTF-8. It's down to 9.5KB vs 10KB after gzipping.
- jharsman 16y agoI'm not surprised it doesn't make much of a difference with HTML since the markup contains so much ASCII. But 25 vs 18 kB is almost 40%. That might not be insignificant depending on how much text you're storing. But it's a nitpick really, I just thought it he should have noted some of the disadvantages to UTF-8 as well.