3 ms·
It doesn't really matter since most document formats already use compression. No matter what encoding you use, the amount of entropy is the same. So UTF-8 usual
by kijin 15y ago
It doesn't really matter since most document formats already use compression. No matter what encoding you use, the amount of entropy is the same. So UTF-8 usually compresses better than 2-byte encodings for CJK languages. This nearly compensates for the increased size.
Example: The Korean text of the Universal Declaration of Human Rights [1] is 8.1KB in EUC-KR and 11.2KB in UTF-8. When compressed with bzip2, it's only 3.1KB and 3.2KB, respectively. I assume Japanese would behave similarly.
[1] http://www.ohchr.org/EN/UDHR/Pages/Language.aspx?LangID=kkn http://www.ohchr.org/EN/UDHR/Pages/Language.aspx?LangID=kkn