3 ms·
The author compares UTF-8 to UTF-16, while there are a myriad better encodings than both for different Asian languages. For instance: EUC-JP in Japan, BIG5 in
by muyuu 15y ago
The author compares UTF-8 to UTF-16, while there are a myriad better encodings than both for different Asian languages.
For instance: EUC-JP in Japan, BIG5 in Taiwan, GB in China, etc. Different EUC encodings are variable-length and are a lot more efficient since they put common subsets in lower parts of the table for each language, close to each other so they also compress better, while allowing tricks for text matching and searches (not really necessary for web sites, but it's nice to use the same encoding throughout applications sometimes). Russian and Greek are also basically multiplied by two in size.
There are a lot of other considerations.
If you think a 30%+ saving in size (and latency) is not a big deal, then you're a lot more likely to lose to local competitors. Note that gzipped or otherwise compressed text makes differences even worse at least in the case of Japanese - where UTF-8 text gets de-aligned all over the place to odd byte sizes and compresses worse. Add to that the fact that Asians browse the net A LOT from the phone and have done so for much longer than westerners and in a bigger percentage, and you have your problem exacerbated even further.
There is a lot more to consider and like it or not it's not as simple as "UTF-8 for everything and everybody, ever!"
- kijin 15y agoHave you ever worked with a system that needs to deal with more than one language at the same time? What if your users want to mix Japanese with Russian in the same sentence? Or Japanese and simplified Chinese? (Yes, people do that.) In the global Internet, UTF-16 and UTF-8 are the only games in town.
- muyuu 15y agoAll the damn time I'm using several languages. Then UTF (and EUC's) are the way to go. It's not like you have to use the same encoding all the time.
- derleth 15y ago> It's not like you have to use the same encoding all the time. Then you are going to feed someone garbage. Why feed people garbage?
- muyuu 15y ago?? Not if you know what you're doing. Not any more than using utf8 exclusively all the time and for all purposes.
- derleth 15y ago> Not if you know what you're doing. This is nice in theory. In practice, people make mistakes. Make it easy on yourself. > Not any more than using utf8 exclusively all the time and for all purposes. Maybe I was unclear: Feeding me Chinese text in UTF-8 is not garbage. Feeding me anything in one of the GB encodings is garbage. Garbage, to me, is text in an encoding I can't handle. If you only use UTF-8, that cannot possibly happen.
- jeltz 15y ago> Note that gzipped or otherwise compressed text makes differences even worse at least in the case of Japanese - where UTF-8 text gets de-aligned all over the place to odd byte sizes and compresses worse. That was not what this article measured. UTF-8 and UTF-16 compressed to virtually the same size.