3 ms·
I think it's also more space efficient when dealing with foreign languages primarily. For plain text, yes. UTF-16 covers the Basic Multilingual Plane in 2 byte
by danieldk 8y ago
I think it's also more space efficient when dealing with foreign languages primarily.
For plain text, yes. UTF-16 covers the Basic Multilingual Plane in 2 bytes per character, UTF-8 covers the BMP in at most 3 bytes per character. However, since most (web) content is dominated by markup in ASCII, which is 1 byte per character in UTF-8 and 2 bytes per character in UTF-16, UTF-8 typically wins even with Asian scripts.
Taking some random zh Wikipedia page in UTF-8:
curl "https://zh.wikipedia.org/wiki/威蘭運河" | wc -c
78543
curl "https://zh.wikipedia.org/wiki/威蘭運河" | wc -m
71003
So, that's 1.11 bytes per character on average, compared to at least 2 bytes per character if it was encoded in UTF-16.
(Of course, adding compression is going to diminish the differences anyway. A very primitive compression algorithm in such cases would be recoding your UTF-16 to UTF-8 :p.)
- Symbiote 8y agocurl -Ss "https://zh.wikipedia.org/wiki/威蘭運河" | iconv -t utf-16 | wc -c 142008 There are only 3799 non-ASCII characters on that page: curl -Ss "https://zh.wikipedia.org/wiki/威蘭運河" | tr -d '\1-\176' | wc -m 3799 Compared with 67204 ASCII ones (tr -cd).
- danieldk 8y agoThe 142008 is to be expected. The page has 71003 code points, 2 bytes per code point (I guess that zh.wikipedia.org is simplified Chinese and normally only uses characters from the BMP) gives 142006 bytes. I'd guess that the two remaining bytes are the byte order mark.