5 ms·

What's your reasoning? Where there is significant markup, UTF-16 encoding is rarely more space-efficient than UTF-8. When compression is involved, there is rar

What's your reasoning? Where there is significant markup, UTF-16 encoding is rarely more space-efficient than UTF-8. When compression is involved, there is rarely a difference.

thaumasiotes 7y ago
UTF-16 may not be great, but UTF-8 is an obviously bad choice for Asian languages. UTF-8 is a variable-width encoding on the following system: ASCII: 1 byte European characters: 2 bytes Asian characters: 3 bytes There is no reason you'd want this in Asia. China uses GB2312, which flips it around: ASCII: 1 byte Asian characters: 2 bytes European characters: 3 bytes
- jhanschoo 7y ago
  This doesn't address my points. That where there is markup, the additional space required by the ASCII markup on UTF16 may very well offset the space savings of Asian text in UTF8. And also, that with compression, these size arguments become negligible for any sufficiently long document written in one language.
  - thaumasiotes 7y ago
    I conceded that with "UTF-16 may not be great". I'm responding to the idea that UTF-8 is such a good solution to the problem that everyone should just use UTF-8. It isn't.
    - hsivonen 7y ago
      For writing globally appropriate software, UTF-8 is the best option that exists in practice. Note that “Asian” is quite a gamut in terms of UTF-8 size: Chinese is at the very compact end of the spectrum and the Burmese script at the other end: https://hsivonen.fi/string-length/#counts https://hsivonen.fi/string-length/#counts
      - camgunz 7y ago
        Wow what a resource, I love this. I'm a little skeptical about that table though, because as you point out, there's not a great way of figuring "meaning per character". For some more (real flawed) comparison, the English version of The Tale of Genji is ~60k words and 224 pages, and the Chinese version is ~75k words and 300 pages. [1] [2] So yeah, I'm skeptical about the claim that some languages have more meaning per character, and as a result you'll end up storing less text overall. I think it'd be cool to look at more data about this though, like (for example) stats from Treasure Data [3]. But regardless, it's pretty indisputable that UTF-16 is a lot better space-wise for languages that tend to be multibyte in UTF-8. Mostly what I'm trying (Quixotically) to say is "UTF-8 the world" ignores a lot of the world. [1]: https://www.readinglength.com/book/isbn-4805314648 https://www.readinglength.com/book/isbn-4805314648 [2]: https://www.readinglength.com/book/isbn-7544717275 https://www.readinglength.com/book/isbn-7544717275 [3]: https://www.treasuredata.com/ https://www.treasuredata.com/
        thaumasiotes 7y ago
        > the Chinese version is ~75k words and 300 pages. [1] [2] Note that link [2] says "guess [of # of words] based on page count". 75k words, 300 pages is one statistic, not two separate statistics. (Estimating words based on page count is likely to be highly reliable, but still.) > I'm skeptical about the claim that some languages have more meaning per character, and as a result you'll end up storing less text overall. Your skepticism is unwarranted. From a pure information-theoretic perspective, the claim that some languages have more meaning per character is a slam dunk, and less than a second of examination proves it conclusively. For example, an entire English novel is unlikely to use more than 256 unique characters. A Chinese novel couldn't use anywhere near that few without sounding incredibly artificial. Here are some single-character words in modern Mandarin: 高 - high/tall 低 - low 大 - big 小 - small 最 - most (superlative marker) 更 - more (comparative marker) 到 - arrive 走 - leave (go away) 玩 - play (e.g. a game) 看 - look 听 - listen 贵 - expensive 爱 - love (verb) 恨 - hate (verb) 龍 - dragon For something more representative, here are some song lyrics -- I'll enclose every word of multiple characters. Unenclosed stretches of text are words of one character each: 都怪我不[小心]地 [偷看了]你的[眼睛] [迷失了][自己] [不知][为什么][欢喜] [期待]听你的[声音] 是[那么][甜蜜] 我[拿着][电话]在你家门外想对你[说出]我[内心]中对你的[期待] [忽然][感觉][浑身][热血][澎湃] [今天]我[想要][[鼓起][勇气]][大声][表白] How many distinct one-character words do you think English could practically support? How many twos? In this verse-and-a-half, there's one word that's always[1] three characters and two that have reached three characters by picking up a verb suffix, for a total of three words that are longer than two characters. (鼓起勇气 is kind of a special case, in that it's a well-known fixed expression, but its meaning is transparent as an ordinary combination of the two words 鼓起 and 勇气, which also see use outside the expression.) [1] Actually, 什么 ["what"] has the vernacular contraction 啥, and this also applies to 为什么 ["why"], so you could argue that the word is sometimes just two characters.
- SAI_Peregrinus 7y ago
  Yet the higher information density of Chinese & Japanese means the total file size tends to be about the same. Korean gets a bit short-changed, and some of the Indian subcontinent languages have low-density (like Latin) but high size (in with the Asian characters) and lose out.
camgunz 7y ago
There's a lot of text out there that isn't markup, or where markup is a sparse minority. Consider anything binary, ebooks, PDFs, various document formats, almost anything in a database, almost all messages sent via messaging apps, almost all emails, any JSON, anything in a protobuf, etc. etc. The vast majority of "text" in the world isn't in HTML/XML. > When compression is involved, there is rarely a difference. Rare is the situation where smaller input to a compression algorithm leads to larger output, and UTF-16 is smaller in lots of languages. It might rarely make a difference you, but that's not the same as rarely making a difference.

What's your reasoning? Where there is significant markup, UTF-16 encoding is rarely more space-efficient than UTF-8. When compression is involved, there is rarely a difference.

thaumasiotes 7y ago

UTF-16 may not be great, but UTF-8 is an obviously bad choice for Asian languages. UTF-8 is a variable-width encoding on the following system: ASCII: 1 byte European characters: 2 bytes Asian characters: 3 bytes There is no reason you'd want this in Asia. China uses GB2312, which flips it around: ASCII: 1 byte Asian characters: 2 bytes European characters: 3 bytes

jhanschoo 7y ago
This doesn't address my points. That where there is markup, the additional space required by the ASCII markup on UTF16 may very well offset the space savings of Asian text in UTF8. And also, that with compression, these size arguments become negligible for any sufficiently long document written in one language.
- thaumasiotes 7y ago
  I conceded that with "UTF-16 may not be great". I'm responding to the idea that UTF-8 is such a good solution to the problem that everyone should just use UTF-8. It isn't.
  - hsivonen 7y ago
    For writing globally appropriate software, UTF-8 is the best option that exists in practice. Note that “Asian” is quite a gamut in terms of UTF-8 size: Chinese is at the very compact end of the spectrum and the Burmese script at the other end: https://hsivonen.fi/string-length/#counts https://hsivonen.fi/string-length/#counts
    - camgunz 7y ago
      Wow what a resource, I love this. I'm a little skeptical about that table though, because as you point out, there's not a great way of figuring "meaning per character". For some more (real flawed) comparison, the English version of The Tale of Genji is ~60k words and 224 pages, and the Chinese version is ~75k words and 300 pages. [1] [2] So yeah, I'm skeptical about the claim that some languages have more meaning per character, and as a result you'll end up storing less text overall. I think it'd be cool to look at more data about this though, like (for example) stats from Treasure Data [3]. But regardless, it's pretty indisputable that UTF-16 is a lot better space-wise for languages that tend to be multibyte in UTF-8. Mostly what I'm trying (Quixotically) to say is "UTF-8 the world" ignores a lot of the world. [1]: https://www.readinglength.com/book/isbn-4805314648 https://www.readinglength.com/book/isbn-4805314648 [2]: https://www.readinglength.com/book/isbn-7544717275 https://www.readinglength.com/book/isbn-7544717275 [3]: https://www.treasuredata.com/ https://www.treasuredata.com/
      - thaumasiotes 7y ago
        > the Chinese version is ~75k words and 300 pages. [1] [2] Note that link [2] says "guess [of # of words] based on page count". 75k words, 300 pages is one statistic, not two separate statistics. (Estimating words based on page count is likely to be highly reliable, but still.) > I'm skeptical about the claim that some languages have more meaning per character, and as a result you'll end up storing less text overall. Your skepticism is unwarranted. From a pure information-theoretic perspective, the claim that some languages have more meaning per character is a slam dunk, and less than a second of examination proves it conclusively. For example, an entire English novel is unlikely to use more than 256 unique characters. A Chinese novel couldn't use anywhere near that few without sounding incredibly artificial. Here are some single-character words in modern Mandarin: 高 - high/tall 低 - low 大 - big 小 - small 最 - most (superlative marker) 更 - more (comparative marker) 到 - arrive 走 - leave (go away) 玩 - play (e.g. a game) 看 - look 听 - listen 贵 - expensive 爱 - love (verb) 恨 - hate (verb) 龍 - dragon For something more representative, here are some song lyrics -- I'll enclose every word of multiple characters. Unenclosed stretches of text are words of one character each: 都怪我不[小心]地 [偷看了]你的[眼睛] [迷失了][自己] [不知][为什么][欢喜] [期待]听你的[声音] 是[那么][甜蜜] 我[拿着][电话]在你家门外想对你[说出]我[内心]中对你的[期待] [忽然][感觉][浑身][热血][澎湃] [今天]我[想要][[鼓起][勇气]][大声][表白] How many distinct one-character words do you think English could practically support? How many twos? In this verse-and-a-half, there's one word that's always[1] three characters and two that have reached three characters by picking up a verb suffix, for a total of three words that are longer than two characters. (鼓起勇气 is kind of a special case, in that it's a well-known fixed expression, but its meaning is transparent as an ordinary combination of the two words 鼓起 and 勇气, which also see use outside the expression.) [1] Actually, 什么 ["what"] has the vernacular contraction 啥, and this also applies to 为什么 ["why"], so you could argue that the word is sometimes just two characters.
SAI_Peregrinus 7y ago
Yet the higher information density of Chinese & Japanese means the total file size tends to be about the same. Korean gets a bit short-changed, and some of the Indian subcontinent languages have low-density (like Latin) but high size (in with the Asian characters) and lose out.

camgunz 7y ago

There's a lot of text out there that isn't markup, or where markup is a sparse minority. Consider anything binary, ebooks, PDFs, various document formats, almost anything in a database, almost all messages sent via messaging apps, almost all emails, any JSON, anything in a protobuf, etc. etc. The vast majority of "text" in the world isn't in HTML/XML. > When compression is involved, there is rarely a difference. Rare is the situation where smaller input to a compression algorithm leads to larger output, and UTF-16 is smaller in lots of languages. It might rarely make a difference you, but that's not the same as rarely making a difference.