6 ms·

What's depressing about the saga of utf8 adoption is that it's such an obvious solution. The ISO2022 standards used the principle since the 70s, and programmers

What's depressing about the saga of utf8 adoption is that it's such an obvious solution. The ISO2022 standards used the principle since the 70s, and programmers of the era were widely accustomed to extending 8 bit instruction sets to multibyte instructions. It's one of those cases where you need to hit your head hard and repeatedly until you bite the bullet and accept that the cost of multibyte (for example, strlen() different from character count, need to parse to seek etc.) are trivial compared to the benefits.

aew4ytasghe5 4y ago
UTF8 is only superior in encoding the american keyboard (7 bits of ascii). All the latin letters map to 1 byte encodings. This is why UTF8 wasn't universally hailed to begin with. Countries like Japan had already solved encoding differently while US was still using ASCII. (See JIS encoding). As time passed tho, the byte savings matter less. Also combined with transfer compression makes the encoding size differences negligible.
- Beltalowda 4y ago
  > UTF8 is only superior in encoding the american keyboard (7 bits of ascii). All the latin letters map to 1 byte encodings. In practice, basic 7-bit ASCII characters dominate a lot of texts, even those in East-Asian languages, as all the markup and such around the actual text is almost always in plain ASCII. For example, if I request a Japanese document from a HTTP server then the HTTP headers, HTML tags, CSS is all in single-byte characters. Of course, there are still many scenarios where UTF-16 "wins" in terms of size – especially with plain text – but real-world size comparisons tend to be a bit more tricky. For example, for the current Wikipedia homepage of ja, zh, and ko: % file * ja-16.htm: HTML document, Unicode text, UTF-16, little-endian text, with very long lines (2709) ja-8.htm: HTML document, Unicode text, UTF-8 text, with very long lines (2709) ko-16.htm: HTML document, Unicode text, UTF-16, little-endian text, with very long lines (2503) ko-8.htm: HTML document, Unicode text, UTF-8 text, with very long lines (2656) zh-16.htm: HTML document, Unicode text, UTF-16, little-endian text, with very long lines (5589) zh-8.htm: HTML document, Unicode text, UTF-8 text, with very long lines (5589) % ls -lh 210K ja-16.htm 118K ja-8.htm 193K ko-16.htm 106K ko-8.htm 189K zh-16.htm 104K zh-8.htm The UTF-16 versions are all larger (and that's just the HTML, excluding the CSS, JS, etc). I wrote a small script to count the number of "wide" multibyte characters: ja-8.htm 100,808 7-bit characters; 19,026 wide characters ko-8.htm 93,888 7-bit characters; 14,149 wide characters zh-8.htm 91,589 7-bit characters; 14,360 wide characters It was higher than I expected, although not that surprising when looking at the source when you have things like: <div class="vector-page-toolbar"> <div class="vector-page-toolbar-container"> <div id="left-navigation"> <nav aria-label="名前空間">
  - knolax 4y ago
    Your argument uses a website whose code was written by English speakers. There would still be ASCII but verbose element names like "vector-page-toolbar-container" would definitely be shorter both in utf-8 and in utf-16 if they weren't written in English
    - Beltalowda 4y ago
      <div class=..> is still <div class=..>, no matter the language of the author. As is "background-color: #fff" in CSS. The UTF-16 is almost double the size, so you'd have to replace a lot of identifiers with 2 or 3 byte ones. Plus many identifiers come from libraries, and when creating their own identifiers many people use either full English or partial English no matter what language (it was a huge mistake to not use English for many identifiers in my first programming job, as you will invariably end up with a mishmash of two languages). But it is easy enough to verify this with some actual websites: https://www.rakuten.co.jp https://www.rakuten.co.jp is 330K in UTF-8 and 625K in UTF-16, https://ameblo.jp https://ameblo.jp is 104K in UTF-8 and 187K in UTF-16, baidu.com is 360K in UTF-8 and 717K in UTF-16, sina.com.cn: 455K, 854K, daum.net: 666K, 1.2M. And all of that is only the HTML document; if we'd add up the CSS – where there's almost no possibility to use non-ASCII outside of class and ID names – and JavaScript – where the filesize is usually dominated by React or jQuery or whatnot – things would skew even more in favour of UTF-8. I'm sure there are examples where a page served over UTF-16 is smaller, such as pages with very little markup (like e.g. HN), but that is the common case, even for websites exclusively written and for users of CJK languages. Someone who does not speak a word of English will save many bytes of data every day with UTF-8. There's a reason all those websites are served over UTF-8 and not UTF-16. But for the sake of the argument, let's replace all class="...", id="..", and data-event-name=".." with strings of the same length consisting of "回". That grows the filesize from 118K to 151K, and ... it's still larger in UTF-16 with 207K. We could start replacing more stuff and eventually UTF-16 may win, maybe. But you have to use a lot of CJK. Let's use a random excerpt: <button id="回回回回回回回回回回回回回回回回回回回回回回回回" tabindex="-1" data-event-name="回回回回回回回回回回回回回回回回回回回回回回回回回回回回回回回回回回回回" Has 91 7-bit characters and 60 multibyte ones (this includes indentation, which may not be represented 100% accurately here). If we do the math this is: UTF-8 91×1 + 60×3 = 271 bytes UTF-16 91×2 + 60×2 = 302 bytes UTF-8 still wins. And to repeat, there are certainly cases where UTF-16 is smaller. Markdown documents and other plain text files is an obvious one, but HTML is rarely one of them. But imagine actually checking things before making a claim...
- msla 4y ago
  UTF-8 is superior due to self-synchronizing features and the fact truncated or otherwise damaged sequences are obviously damaged, even if they use unknown characters.
fomine3 4y ago
ISO 2022 isn't beautiful solution like UTF-8 self sync encoding. Backward compatibility is also great (so called UTF FSS). It was designed by smart Plan9 people.
- manholio 4y ago
  If Ken and Pike could design it in an afternoon, I'm sure even a standards committee could have done it in less than 5 years or so, the ideas are straight forward. I would also argue that the fast sync with at most one character consumed is a good feature but not essential for adoption. Any byte-oriented Unicode packing scheme that can gracefully consume ASCII is better than multibyte, codepages sent out-of-band etc. etc.

aew4ytasghe5 4y ago

UTF8 is only superior in encoding the american keyboard (7 bits of ascii). All the latin letters map to 1 byte encodings. This is why UTF8 wasn't universally hailed to begin with. Countries like Japan had already solved encoding differently while US was still using ASCII. (See JIS encoding). As time passed tho, the byte savings matter less. Also combined with transfer compression makes the encoding size differences negligible.

Beltalowda 4y ago
> UTF8 is only superior in encoding the american keyboard (7 bits of ascii). All the latin letters map to 1 byte encodings. In practice, basic 7-bit ASCII characters dominate a lot of texts, even those in East-Asian languages, as all the markup and such around the actual text is almost always in plain ASCII. For example, if I request a Japanese document from a HTTP server then the HTTP headers, HTML tags, CSS is all in single-byte characters. Of course, there are still many scenarios where UTF-16 "wins" in terms of size – especially with plain text – but real-world size comparisons tend to be a bit more tricky. For example, for the current Wikipedia homepage of ja, zh, and ko: % file * ja-16.htm: HTML document, Unicode text, UTF-16, little-endian text, with very long lines (2709) ja-8.htm: HTML document, Unicode text, UTF-8 text, with very long lines (2709) ko-16.htm: HTML document, Unicode text, UTF-16, little-endian text, with very long lines (2503) ko-8.htm: HTML document, Unicode text, UTF-8 text, with very long lines (2656) zh-16.htm: HTML document, Unicode text, UTF-16, little-endian text, with very long lines (5589) zh-8.htm: HTML document, Unicode text, UTF-8 text, with very long lines (5589) % ls -lh 210K ja-16.htm 118K ja-8.htm 193K ko-16.htm 106K ko-8.htm 189K zh-16.htm 104K zh-8.htm The UTF-16 versions are all larger (and that's just the HTML, excluding the CSS, JS, etc). I wrote a small script to count the number of "wide" multibyte characters: ja-8.htm 100,808 7-bit characters; 19,026 wide characters ko-8.htm 93,888 7-bit characters; 14,149 wide characters zh-8.htm 91,589 7-bit characters; 14,360 wide characters It was higher than I expected, although not that surprising when looking at the source when you have things like: <div class="vector-page-toolbar"> <div class="vector-page-toolbar-container"> <div id="left-navigation"> <nav aria-label="名前空間">
- knolax 4y ago
  Your argument uses a website whose code was written by English speakers. There would still be ASCII but verbose element names like "vector-page-toolbar-container" would definitely be shorter both in utf-8 and in utf-16 if they weren't written in English
  - Beltalowda 4y ago
    <div class=..> is still <div class=..>, no matter the language of the author. As is "background-color: #fff" in CSS. The UTF-16 is almost double the size, so you'd have to replace a lot of identifiers with 2 or 3 byte ones. Plus many identifiers come from libraries, and when creating their own identifiers many people use either full English or partial English no matter what language (it was a huge mistake to not use English for many identifiers in my first programming job, as you will invariably end up with a mishmash of two languages). But it is easy enough to verify this with some actual websites: https://www.rakuten.co.jp https://www.rakuten.co.jp is 330K in UTF-8 and 625K in UTF-16, https://ameblo.jp https://ameblo.jp is 104K in UTF-8 and 187K in UTF-16, baidu.com is 360K in UTF-8 and 717K in UTF-16, sina.com.cn: 455K, 854K, daum.net: 666K, 1.2M. And all of that is only the HTML document; if we'd add up the CSS – where there's almost no possibility to use non-ASCII outside of class and ID names – and JavaScript – where the filesize is usually dominated by React or jQuery or whatnot – things would skew even more in favour of UTF-8. I'm sure there are examples where a page served over UTF-16 is smaller, such as pages with very little markup (like e.g. HN), but that is the common case, even for websites exclusively written and for users of CJK languages. Someone who does not speak a word of English will save many bytes of data every day with UTF-8. There's a reason all those websites are served over UTF-8 and not UTF-16. But for the sake of the argument, let's replace all class="...", id="..", and data-event-name=".." with strings of the same length consisting of "回". That grows the filesize from 118K to 151K, and ... it's still larger in UTF-16 with 207K. We could start replacing more stuff and eventually UTF-16 may win, maybe. But you have to use a lot of CJK. Let's use a random excerpt: <button id="回回回回回回回回回回回回回回回回回回回回回回回回" tabindex="-1" data-event-name="回回回回回回回回回回回回回回回回回回回回回回回回回回回回回回回回回回回回" Has 91 7-bit characters and 60 multibyte ones (this includes indentation, which may not be represented 100% accurately here). If we do the math this is: UTF-8 91×1 + 60×3 = 271 bytes UTF-16 91×2 + 60×2 = 302 bytes UTF-8 still wins. And to repeat, there are certainly cases where UTF-16 is smaller. Markdown documents and other plain text files is an obvious one, but HTML is rarely one of them. But imagine actually checking things before making a claim...
msla 4y ago
UTF-8 is superior due to self-synchronizing features and the fact truncated or otherwise damaged sequences are obviously damaged, even if they use unknown characters.

fomine3 4y ago

ISO 2022 isn't beautiful solution like UTF-8 self sync encoding. Backward compatibility is also great (so called UTF FSS). It was designed by smart Plan9 people.

manholio 4y ago
If Ken and Pike could design it in an afternoon, I'm sure even a standards committee could have done it in less than 5 years or so, the ideas are straight forward. I would also argue that the fast sync with at most one character consumed is a good feature but not essential for adoption. Any byte-oriented Unicode packing scheme that can gracefully consume ASCII is better than multibyte, codepages sent out-of-band etc. etc.