5 ms·

I feel like we're at a point now where articles that just try to 'demystify' unicode are almost teaching the controversy if they don't come out and actually say

I feel like we're at a point now where articles that just try to 'demystify' unicode are almost teaching the controversy if they don't come out and actually say how you should deal with encodings in new apps. It's about time we actually start pressing for the idea that utf-16 was a terrible idea and that utf-8 should be the dominant wire format for unicode, with ucs4 if you really need to have a linear representation. Utf-16 is confusing, complicated, and implementations are routinely broken because the corner cases are rarer. I really hope we're not still stuck with it in 50 years.

salmonellaeater 13y ago
What is the argument for utf-8 that doesn't also apply to utf-16, depending on what your common characters are? Applications that use Asian languages take a big hit from utf-8.
- querulous 13y ago
  there are no surrogates in utf-8. that alone is reason to use it over utf-16
  - millstone 13y ago
    I don't get it. Surrogates in UTF-16 are when you have two code units that map to a single code point. In UTF-8, you have that same issue, except it may be two, three, or four code units. So how is UTF-8 better in this respect?
- Dylan16807 13y ago
  >What is the argument for utf-8 that doesn't also apply to utf-16 The comment you're replying to says right there, "implementations are routinely broken because the corner cases are rarer" utf-16 misleads people into using fixed-width code where it won't work right. >Applications that use Asian languages take a big hit from utf-8. Not really. It's a moderate hit for pure text, but how often do you see bulk pure text, without any markup, and that can't be compressed?
  - salmonellaeater 13y ago
    > utf-16 misleads people into using fixed-width code where it won't work right This is exactly my point: lots of code also treats utf-8 as if it were ASCII. > how often do you see bulk pure text, without any markup, and that can't be compressed? That's a good argument. Thanks :)
    - stormbrew 13y ago
      > This is exactly my point: lots of code also treats utf-8 as if it were ASCII. Two things to this: - In many cases this is an entirely non-destructive mistake to make. This is an advantage of utf-8, that intermediaries that only care about ascii characters (even in the presence of extended characters) can work with it. - It will become readily apparent that you've got a problem on the first attempt to port your app to any language with a non-ascii script. With UTF-16 it only becomes apparent when you go beyond the BMP.
aerolite 13y ago
More ethnocentrism. UTF-16 results in less bandwidth for any language that has non-ASCII characters.
- pedrocr 13y ago
  That's not true. For anything that uses the latin alphabet utf8 will be strictly better as the majority of characters will be in ASCII (1 byte) and the rest will just need an extra byte, equaling utf16. Looking at the format definition[1] anything below U+0800 will fit in 2 bytes in UTF8. So the ethnocentrism starts at Samaritan[2]. The Japanese, Chinese and Indian scripts are probably the most important in that range. [1] https://en.wikipedia.org/wiki/UTF-8#Description https://en.wikipedia.org/wiki/UTF-8#Description [2] http://codepoints.net/basic_multilingual_plane http://codepoints.net/basic_multilingual_plane
  - gsnedders 13y ago
    I can't find the research hsivonen (probably best known for validator.nu and Gecko's HTML parser) did into this a while ago, but he concluded that for the top 100 (IIRC) CJK sites, they were smaller as gzip'd UTF-8 than gzip'd UTF-16, because they all contained a sufficient quantity of HTML/CSS/JS that the gain from the shorter representation of them outweighed that of the text itself.
- stormbrew 13y ago
  I think the Unicode group's handling of the round trip issues faced by Japanese and Chinese users with their own encodings due to Han Unification is a bigger ethnocentric problem by far. I'm not unaware of these issues. But it's not like utf-16 was invented to solve that problem so much as it was an accident that in the early days of unicode 2 bytes was enough to encode all code points and a 2 byte encoding was appealing as a universal solution in a way that a 3 or 4 byte encoding never would be. Some group of people on the planet will have to suffer 3 or more byte encodings no matter what. And I'd be happy to make it mine if it meant we could go with the better technical solution, if I had the power to do that. All that aside it's certainly not quite as clear cut as any language. Many languages have a mix of ascii and non-ascii characters and those fare better than they would with utf-16. And in the codepoints below 0x07ff (where utf-8 goes up to 3 bytes) there are character sets like greek, hebrew, arabic, and cyrillic. And as has been pointed out, ascii characters are often used in structural elements of documents and this factors into making it unlikely that the general case is as bad as the worst case (50% inflation) for most practical text. Particularly when you throw compression into the mix. [edit] fixed some minor things.
millstone 13y ago
I agree about wire formats. However, programming languages that use UTF-8 as the primary string representation tend to have inferior Unicode support to those that use UTF-16. I think this is because with UTF-8, a lot of stuff just seems to work and so it's easy to ignore the issues, while UTF-16 forces you to come to grips with the realities of Unicode. For example, consider a function to open a file by name. If your strings are UTF-8, you can just pass your null terminated buffer to fopen() or something and things will work fine for most files. But if your strings are internally UTF-16, you have to think about which encoding to use, and you research it, and you discover that holy crap, this stuff differs across OSes, and so we better take this problem seriously.
- plorkyeran 13y ago
  UTF-16 has the same issue with invalid things appearing to work due to surrogate pairs. A lot of UTF-16 based applications actually only support UCS-2, because you can assume one string index equals one character and get away with it if it never occurs to you to try to use stuff outside of the BMP.
- stormbrew 13y ago
  > However, programming languages that use UTF-8 as the primary string representation tend to have inferior Unicode support to those that use UTF-16. I'm really curious which languages drove you to this conclusion. Right now we're in a state where the most mature implementations of unicode in general are utf-16 because of the ucs2 accident, so it wouldn't exactly surprise me, but I'd still like to see the proof.
SonOfLilit 13y ago
You're probably looking for this: http://www.utf8everywhere.org/ http://www.utf8everywhere.org/

salmonellaeater 13y ago

What is the argument for utf-8 that doesn't also apply to utf-16, depending on what your common characters are? Applications that use Asian languages take a big hit from utf-8.

querulous 13y ago
there are no surrogates in utf-8. that alone is reason to use it over utf-16
- millstone 13y ago
  I don't get it. Surrogates in UTF-16 are when you have two code units that map to a single code point. In UTF-8, you have that same issue, except it may be two, three, or four code units. So how is UTF-8 better in this respect?
Dylan16807 13y ago
>What is the argument for utf-8 that doesn't also apply to utf-16 The comment you're replying to says right there, "implementations are routinely broken because the corner cases are rarer" utf-16 misleads people into using fixed-width code where it won't work right. >Applications that use Asian languages take a big hit from utf-8. Not really. It's a moderate hit for pure text, but how often do you see bulk pure text, without any markup, and that can't be compressed?
- salmonellaeater 13y ago
  > utf-16 misleads people into using fixed-width code where it won't work right This is exactly my point: lots of code also treats utf-8 as if it were ASCII. > how often do you see bulk pure text, without any markup, and that can't be compressed? That's a good argument. Thanks :)
  - stormbrew 13y ago
    > This is exactly my point: lots of code also treats utf-8 as if it were ASCII. Two things to this: - In many cases this is an entirely non-destructive mistake to make. This is an advantage of utf-8, that intermediaries that only care about ascii characters (even in the presence of extended characters) can work with it. - It will become readily apparent that you've got a problem on the first attempt to port your app to any language with a non-ascii script. With UTF-16 it only becomes apparent when you go beyond the BMP.

aerolite 13y ago

More ethnocentrism. UTF-16 results in less bandwidth for any language that has non-ASCII characters.

pedrocr 13y ago
That's not true. For anything that uses the latin alphabet utf8 will be strictly better as the majority of characters will be in ASCII (1 byte) and the rest will just need an extra byte, equaling utf16. Looking at the format definition[1] anything below U+0800 will fit in 2 bytes in UTF8. So the ethnocentrism starts at Samaritan[2]. The Japanese, Chinese and Indian scripts are probably the most important in that range. [1] https://en.wikipedia.org/wiki/UTF-8#Description https://en.wikipedia.org/wiki/UTF-8#Description [2] http://codepoints.net/basic_multilingual_plane http://codepoints.net/basic_multilingual_plane
- gsnedders 13y ago
  I can't find the research hsivonen (probably best known for validator.nu and Gecko's HTML parser) did into this a while ago, but he concluded that for the top 100 (IIRC) CJK sites, they were smaller as gzip'd UTF-8 than gzip'd UTF-16, because they all contained a sufficient quantity of HTML/CSS/JS that the gain from the shorter representation of them outweighed that of the text itself.
stormbrew 13y ago
I think the Unicode group's handling of the round trip issues faced by Japanese and Chinese users with their own encodings due to Han Unification is a bigger ethnocentric problem by far. I'm not unaware of these issues. But it's not like utf-16 was invented to solve that problem so much as it was an accident that in the early days of unicode 2 bytes was enough to encode all code points and a 2 byte encoding was appealing as a universal solution in a way that a 3 or 4 byte encoding never would be. Some group of people on the planet will have to suffer 3 or more byte encodings no matter what. And I'd be happy to make it mine if it meant we could go with the better technical solution, if I had the power to do that. All that aside it's certainly not quite as clear cut as any language. Many languages have a mix of ascii and non-ascii characters and those fare better than they would with utf-16. And in the codepoints below 0x07ff (where utf-8 goes up to 3 bytes) there are character sets like greek, hebrew, arabic, and cyrillic. And as has been pointed out, ascii characters are often used in structural elements of documents and this factors into making it unlikely that the general case is as bad as the worst case (50% inflation) for most practical text. Particularly when you throw compression into the mix. [edit] fixed some minor things.

millstone 13y ago

I agree about wire formats. However, programming languages that use UTF-8 as the primary string representation tend to have inferior Unicode support to those that use UTF-16. I think this is because with UTF-8, a lot of stuff just seems to work and so it's easy to ignore the issues, while UTF-16 forces you to come to grips with the realities of Unicode. For example, consider a function to open a file by name. If your strings are UTF-8, you can just pass your null terminated buffer to fopen() or something and things will work fine for most files. But if your strings are internally UTF-16, you have to think about which encoding to use, and you research it, and you discover that holy crap, this stuff differs across OSes, and so we better take this problem seriously.

plorkyeran 13y ago
UTF-16 has the same issue with invalid things appearing to work due to surrogate pairs. A lot of UTF-16 based applications actually only support UCS-2, because you can assume one string index equals one character and get away with it if it never occurs to you to try to use stuff outside of the BMP.
stormbrew 13y ago
> However, programming languages that use UTF-8 as the primary string representation tend to have inferior Unicode support to those that use UTF-16. I'm really curious which languages drove you to this conclusion. Right now we're in a state where the most mature implementations of unicode in general are utf-16 because of the ucs2 accident, so it wouldn't exactly surprise me, but I'd still like to see the proof.

SonOfLilit 13y ago

You're probably looking for this: http://www.utf8everywhere.org/ http://www.utf8everywhere.org/