4 ms·
Tell that to the Japanese, whose glyphs are always 3+ bytes in UTF-8!
by garethadams 15y ago
Tell that to the Japanese, whose glyphs are always 3+ bytes in UTF-8!
- mekoka 15y agoVery astute point and I would grant that it could be a consideration when writing a program for which you don't intend to ever output content on the web. I personally believe that an extra byte per character is a rather small price to pay for all the good of utf-8 and I would not even give a second look to utf-16. Now, consider that if you decide to use utf-16 to publish your Japanese text as an HTML document (whose syntax is entirely ascii based), you might lose its "saving advantage" and could easily end up with text that is actually larger in utf-16 than in utf-8. Also take a look at this: http://programmers.stackexchange.com/questions/102205/should-utf-16-be-considered-harmful http://programmers.stackexchange.com/questions/102205/should...
- lubutu 15y agoI'm curious whether there is more information per byte in UTF-8 hanzi / kanji compared to English, considering each character is about 3 bytes but can represent an entire word. A potential human-readable compression technique! ;)
- kijin 15y agoIt doesn't really matter since most document formats already use compression. No matter what encoding you use, the amount of entropy is the same. So UTF-8 usually compresses better than 2-byte encodings for CJK languages. This nearly compensates for the increased size. Example: The Korean text of the Universal Declaration of Human Rights [1] is 8.1KB in EUC-KR and 11.2KB in UTF-8. When compressed with bzip2, it's only 3.1KB and 3.2KB, respectively. I assume Japanese would behave similarly. [1] http://www.ohchr.org/EN/UDHR/Pages/Language.aspx?LangID=kkn http://www.ohchr.org/EN/UDHR/Pages/Language.aspx?LangID=kkn