4 ms·

What is the argument for utf-8 that doesn't also apply to utf-16, depending on what your common characters are? Applications that use Asian languages take a big

What is the argument for utf-8 that doesn't also apply to utf-16, depending on what your common characters are? Applications that use Asian languages take a big hit from utf-8.

querulous 13y ago
there are no surrogates in utf-8. that alone is reason to use it over utf-16
- millstone 13y ago
  I don't get it. Surrogates in UTF-16 are when you have two code units that map to a single code point. In UTF-8, you have that same issue, except it may be two, three, or four code units. So how is UTF-8 better in this respect?
Dylan16807 13y ago
>What is the argument for utf-8 that doesn't also apply to utf-16 The comment you're replying to says right there, "implementations are routinely broken because the corner cases are rarer" utf-16 misleads people into using fixed-width code where it won't work right. >Applications that use Asian languages take a big hit from utf-8. Not really. It's a moderate hit for pure text, but how often do you see bulk pure text, without any markup, and that can't be compressed?
- salmonellaeater 13y ago
  > utf-16 misleads people into using fixed-width code where it won't work right This is exactly my point: lots of code also treats utf-8 as if it were ASCII. > how often do you see bulk pure text, without any markup, and that can't be compressed? That's a good argument. Thanks :)
  - stormbrew 13y ago
    > This is exactly my point: lots of code also treats utf-8 as if it were ASCII. Two things to this: - In many cases this is an entirely non-destructive mistake to make. This is an advantage of utf-8, that intermediaries that only care about ascii characters (even in the presence of extended characters) can work with it. - It will become readily apparent that you've got a problem on the first attempt to port your app to any language with a non-ascii script. With UTF-16 it only becomes apparent when you go beyond the BMP.

What is the argument for utf-8 that doesn't also apply to utf-16, depending on what your common characters are? Applications that use Asian languages take a big hit from utf-8.

querulous 13y ago

there are no surrogates in utf-8. that alone is reason to use it over utf-16

millstone 13y ago
I don't get it. Surrogates in UTF-16 are when you have two code units that map to a single code point. In UTF-8, you have that same issue, except it may be two, three, or four code units. So how is UTF-8 better in this respect?

Dylan16807 13y ago

>What is the argument for utf-8 that doesn't also apply to utf-16 The comment you're replying to says right there, "implementations are routinely broken because the corner cases are rarer" utf-16 misleads people into using fixed-width code where it won't work right. >Applications that use Asian languages take a big hit from utf-8. Not really. It's a moderate hit for pure text, but how often do you see bulk pure text, without any markup, and that can't be compressed?

salmonellaeater 13y ago
> utf-16 misleads people into using fixed-width code where it won't work right This is exactly my point: lots of code also treats utf-8 as if it were ASCII. > how often do you see bulk pure text, without any markup, and that can't be compressed? That's a good argument. Thanks :)
- stormbrew 13y ago
  > This is exactly my point: lots of code also treats utf-8 as if it were ASCII. Two things to this: - In many cases this is an entirely non-destructive mistake to make. This is an advantage of utf-8, that intermediaries that only care about ascii characters (even in the presence of extended characters) can work with it. - It will become readily apparent that you've got a problem on the first attempt to port your app to any language with a non-ascii script. With UTF-16 it only becomes apparent when you go beyond the BMP.