4 ms·
It's interesting that UTF-8 has largely achieved its success amongst English speakers and Latin-1 languages--almost everything else has remained more or less st
by gchpaco 17y ago
It's interesting that UTF-8 has largely achieved its success amongst English speakers and Latin-1 languages--almost everything else has remained more or less static or has a slow downward trend. Since we're not seeing an increase in UTF-16 or UCS-2 or anything else like that, this would seem evidence that the Web, or at least Google's view of it, is becoming even more increasingly dominated by Western European languages, which is itself an interesting idea.
- jmillikin 17y agoSince we're not seeing an increase in UTF-16 or UCS-2 or anything else like that, this would seem evidence that the Web, or at least Google's view of it, is becoming even more increasingly dominated by Western European languages, which is itself an interesting idea. This assumes that new pages (in languages with non-Latin scripts) are likely to use national encodings, which (in my experience) is not true. Given any popular* website written in Arabic, Cyrillic, Hangul, Kanji, etc, chances are that it will be in UTF-8. The only times I see encodings like SJIS, KOI8, or ISO-8859-1 any more are in old old pages, written before mass popularity of internet, or on amateur/very small websites. The former were created before UTF-8, the latter by unskilled or inexperienced authors.
- pmjordan 17y agoI don't see how you're inferring language share within UTF-8. Being a unicode encoding, it can be used to represent text in all popular scripts; that's kind of the point.
- gchpaco 17y agoBut UTF-16 is far more advantageous an encoding if you speak Chinese, Arabic, Russian, or Japanese; UTF-8 can take up to four bytes per character for some of those.
- ars 17y agoSo? UTF-16 is a lot more work to support. For example, do I store the BOM in every string when I store a name in a database? It's harder to handle UTF-16 because of all the nulls. And since Chinese stores an entire word in a single character, I'm perfectly fine with their words taking 4 bytes. An average word in English uses 5 letters.
- jerf 17y agoIn a world where I can load my cell phone up with 4GB of flash for ~$10, I don't think anybody really cares. For network transmission, even gzip compression will take the sting out. Oh, I know it's not free. I'm not saying it's free. I'm just saying it's nowhere near the pain point for most people. The days of using a computer and fearing that you might type a report for school so long the machine will run out of memory are long gone.
- jmillikin 17y agoThe interesting character ranges are: U+0000 to U+007F: 1 byte for UTF-8, 2 for UTF-16 U+0080 to U+07FF: 2 bytes for UTF-8 and UTF-16 U+0800 to U+FFFF: 3 bytes for UTF-8, 2 for UTF-16 U+10000 and up: 4 bytes for UTF-8 and UTF-16 Arabic, Cyrillic (Russian), Tamil, Thai, Hebrew, and many other non-European scripts take an equal amount of space in either UTF-8 and UTF-16. Japanese and Korean have some cases where UTF-16 is more efficient, but only by a byte, and most Chinese characters take 4 bytes either way.
- pmjordan 17y agoAdditionally, since we're talking about web pages, the markup tags will still be composed of ASCII characters, for which UTF-8 has an advantage.
- drinian 17y agoCJK adaptation has been slow, not only because I suspect those countries tend to have a more insular Web, but because of the Han unification issues: http://en.wikipedia.org/wiki/Han_unification http://en.wikipedia.org/wiki/Han_unification