5 ms·

ASCII and UTF-8 are too US centric. That's why adoption in places like China is so low. Also, if there's variable length encoding why can't we just do a proper

by alecco 15y ago

ASCII and UTF-8 are too US centric. That's why adoption in places like China is so low. Also, if there's variable length encoding why can't we just do a proper way and improve size for the same computational cost?

paulsutter 15y ago
The author makes a compelling case for UTF-8 in Asian languages. I'd love to hear any specific counter-arguments.
- maaku 15y ago
  No he doesn't, he dismisses it out of hand by choosing an example that is uniquely suited to minimize the advantages of UTF-16 for non-Roman scripts. Precious little of an HTML document is actually textual content.
  - ori_b 15y ago
    I think that's the point; In the real world, any significant chunk of non-roman text is embedded in far more roman text, or in terms of internals of programs, is generally dwarfed by the size of other data structures. More or less, I'd say that storage size of text usually doesn't matter.
  - paulsutter 15y ago
    Precious little of the data stored in the world is textual content. It's hard to believe that choice of encoding standards materially impacts RAM or disk budgets. Most of his points are not related to encoding size but to simplicity and standardization. I find those reasons to be very compelling. I'm asking for clear counterarguments because I concede that my ASCII background could predispose me to UTF-8. Please be a little clearer, I really do want to know.
  - pjscott 15y ago
    HTML documents are hardly unusual examples. Also, look at the other column in that table, where he stripped out the HTML tags and looked only at the body text: UTF-16 was somewhat smaller, and gzipping them made the difference negligible. Does UTF-16 really have such a great advantage for non-Roman writing systems? Or is this motivated more by a disliking for Anglocentrism?
ruediger 15y ago
http://www.utf8everywhere.org/#faq.asians http://www.utf8everywhere.org/#faq.asians
lambda 15y ago
Did you read the article, including the part about Asian text? Like it or not, most text these days is embedded in markup languages like XML or HTML, in which all of the markup is within the ASCII range. This, coupled with the fact that UTF-8 gives you a factor of 2 savings over UTF-16 for the ASCII range, while only a factor of 1.5 increase over UTF-16 for CJK characters, means that for much text (such as anything on the Web), UTF-8 is actually smaller than UTF-16 even for CJK text. Yes, ASCII is obviously too US centric; you can't encode any writing systems other than the Roman alphabet in ASCII. However, that's not at question here. The question is, which Unicode encoding should you use, so you can represent all writing systems in a single encoding. And the major contenders are UTF-8 and UTF-16. The point of this article is, for that purpose, UTF-8 is a far better choice. > Also, if there's variable length encoding why can't we just do a proper way and improve size for the same computational cost? What do you mean by a "proper way"? If size is what you care about, just compress your data. Compression will do a lot better for a much wider range of data than some clever encoding will. UTF-8 is a carefully constructed encoding designed to meet several design criteria. For instance, you could get better size for a wider range of character sets by having a single byte to represent switching between character sets; so you could use that byte, and then a whole bunch of 2 byte CJK characters. But that would defeat one of the design goals of UTF-8, which is to be self synchronizing. That means that if you get a partial sequence (such as a sequence that has been truncated), you can start decoding the characters after a fixed number of bytes. In the case of UTF-8, you will never have to go more than 3 bytes before you can start decoding again. In my hypothetical scheme where certain symbols were used to switch between character sets, you would not be able to interpret anything until you found the next such symbol. This makes UTF-8 more robust in the face of errors. Another design goal of UTF-8 was to be backwards-compatible with ASCII. Like it or not, ASCII has been the standard encoding for decades, and there is a lot of text in ASCII and a lot of software that uses ASCII delimiters and the like. So, while it would be possible, in theory, to define a character encoding that is more "fair" than UTF-8, that ignores many of the other goals of the design of UTF-8. And UTF-8 is widely supported and used (it is the most popular encoding on the Web, even in places like Japan, and a close second in China), while a new encoding would require another large, global, and painful transition process to introduce.
- muyuu 15y ago
alecco 15y ago
The new HN: disagree = downvote
- mokus 15y ago
  It's not really new. Whether you agree with the sentiment or not, pg's oldish comment on the issue at least establishes the existence of the behavior several years ago, and has been taken by many to be the final word on the acceptability of the practice: http://news.ycombinator.com/item?id=117171 http://news.ycombinator.com/item?id=117171 (edit: spelling)
  - alecco 15y ago
    Paired with increasing groupthink, HN discussions are getting boring.

by alecco 15y ago

paulsutter 15y ago

The author makes a compelling case for UTF-8 in Asian languages. I'd love to hear any specific counter-arguments.

maaku 15y ago
No he doesn't, he dismisses it out of hand by choosing an example that is uniquely suited to minimize the advantages of UTF-16 for non-Roman scripts. Precious little of an HTML document is actually textual content.
- ori_b 15y ago
  I think that's the point; In the real world, any significant chunk of non-roman text is embedded in far more roman text, or in terms of internals of programs, is generally dwarfed by the size of other data structures. More or less, I'd say that storage size of text usually doesn't matter.
- paulsutter 15y ago
  Precious little of the data stored in the world is textual content. It's hard to believe that choice of encoding standards materially impacts RAM or disk budgets. Most of his points are not related to encoding size but to simplicity and standardization. I find those reasons to be very compelling. I'm asking for clear counterarguments because I concede that my ASCII background could predispose me to UTF-8. Please be a little clearer, I really do want to know.
- pjscott 15y ago
  HTML documents are hardly unusual examples. Also, look at the other column in that table, where he stripped out the HTML tags and looked only at the body text: UTF-16 was somewhat smaller, and gzipping them made the difference negligible. Does UTF-16 really have such a great advantage for non-Roman writing systems? Or is this motivated more by a disliking for Anglocentrism?

ruediger 15y ago

http://www.utf8everywhere.org/#faq.asians http://www.utf8everywhere.org/#faq.asians

lambda 15y ago

Did you read the article, including the part about Asian text? Like it or not, most text these days is embedded in markup languages like XML or HTML, in which all of the markup is within the ASCII range. This, coupled with the fact that UTF-8 gives you a factor of 2 savings over UTF-16 for the ASCII range, while only a factor of 1.5 increase over UTF-16 for CJK characters, means that for much text (such as anything on the Web), UTF-8 is actually smaller than UTF-16 even for CJK text. Yes, ASCII is obviously too US centric; you can't encode any writing systems other than the Roman alphabet in ASCII. However, that's not at question here. The question is, which Unicode encoding should you use, so you can represent all writing systems in a single encoding. And the major contenders are UTF-8 and UTF-16. The point of this article is, for that purpose, UTF-8 is a far better choice. > Also, if there's variable length encoding why can't we just do a proper way and improve size for the same computational cost? What do you mean by a "proper way"? If size is what you care about, just compress your data. Compression will do a lot better for a much wider range of data than some clever encoding will. UTF-8 is a carefully constructed encoding designed to meet several design criteria. For instance, you could get better size for a wider range of character sets by having a single byte to represent switching between character sets; so you could use that byte, and then a whole bunch of 2 byte CJK characters. But that would defeat one of the design goals of UTF-8, which is to be self synchronizing. That means that if you get a partial sequence (such as a sequence that has been truncated), you can start decoding the characters after a fixed number of bytes. In the case of UTF-8, you will never have to go more than 3 bytes before you can start decoding again. In my hypothetical scheme where certain symbols were used to switch between character sets, you would not be able to interpret anything until you found the next such symbol. This makes UTF-8 more robust in the face of errors. Another design goal of UTF-8 was to be backwards-compatible with ASCII. Like it or not, ASCII has been the standard encoding for decades, and there is a lot of text in ASCII and a lot of software that uses ASCII delimiters and the like. So, while it would be possible, in theory, to define a character encoding that is more "fair" than UTF-8, that ignores many of the other goals of the design of UTF-8. And UTF-8 is widely supported and used (it is the most popular encoding on the Web, even in places like Japan, and a close second in China), while a new encoding would require another large, global, and painful transition process to introduce.

muyuu 15y ago

alecco 15y ago

The new HN: disagree = downvote

mokus 15y ago
It's not really new. Whether you agree with the sentiment or not, pg's oldish comment on the issue at least establishes the existence of the behavior several years ago, and has been taken by many to be the final word on the acceptability of the practice: http://news.ycombinator.com/item?id=117171 http://news.ycombinator.com/item?id=117171 (edit: spelling)
- alecco 15y ago
  Paired with increasing groupthink, HN discussions are getting boring.