4 ms·

Really good article. You'll get nothing from me but heartfelt agreement. I especially liked that the article was giving numbers about how inefficient UTF8 would

by pilif 15y ago

Really good article. You'll get nothing from me but heartfelt agreement. I especially liked that the article was giving numbers about how inefficient UTF8 would be to store Asian text (not really apparently). Also insightful, but obvious in hindsight: Not even in utf-32 you can index specific character in constant time due to the various digraphs. The one property I really love about UTF8 is that you get a free consistency check as not every arbitrary byte sequence is a valid UTF8 string. This is a really good help for detecting encoding errors very early (still to this day, applications are known to lie about the encoding of their output). And of course, there's no endianness issue, removing the need for a BOM which makes it possible for tools that operate at byte levels to still do the right job. If only it had better support outside of Unix. For example, try opening a UTF8 encoded CSV file (using characters outside of ASCII of course) in Mac Excel (latest versions. Up until that, it didn't know UTF8 at all) for a WTF experience somewhere between comical and painful. If there is one thing I could criticize about UTF8 then that would be its similarity to ASCII (which is also its greatest strength) causing many applications and APIs to boldly declare UTF8 compatibility when all they really can do is ASCII compatibility and emitting a mess (or blowing up) once they have to deal with code points outside that range. I'm jokingly calling this US-UTF8 when I encounter it (all too often unfortunately), but maybe the proliferation of "cool" characters like what we recently got with Emoji is likely going to help with this over time.

yuhong 15y ago
Yea, reminds me of DBCS. UTF-8 however don't use bytes below 0x80 as anything other than an ASCII character, unlike some DBCS encodings such as Shift-JIS.
_3u10 15y ago
"The one property I really love about UTF8 is that you get a free consistency check as not every arbitrary byte sequence is a valid UTF8 string." You don't get this at all using UTF-8. You only get it if you attempt to decode the string which even something like strlen doesn't do. Strlen will happily give you wrong answers about how many characters are in a UTF-8 string all day long and never ever attempt to check the validity of the string. Take your valid UTF-8 and change one of the characters to null, now it doesn't work in many circumstances with 'UTF-8' code. Also, should the free consistency check ever actually work you're in a bigger pickle as you now have to figure out whether the string is wrongly encoded UTF-8 or someone sent you extended ASCII. I did a lot of work with unicode apps. I used to have a series of about 5 strings that I could paste into a 'UNICODE' application and have it invariably break. One was an extended ASCII string that happend to be valid UTF-8 sans BOM :) One was a UTF-8 string with BOM and has 0x00 inside :) (I call this string how to tell if it was written with C) One was a UTF-8 string with a BOM :) One UTF-8 string with a some common latin characters, a couple japanese, and a character outside the BMP. Two UTF-16 strings in LE/BE with and sans BOM.
- jsprinkles 15y ago
  > You only get it if you attempt to decode the string which even something like strlen doesn't do. Because strlen() is a count of chars in a null-terminated char[], not a decoder. Ever. It's character set agnostic. > Strlen will happily give you wrong answers about how many characters are in a UTF-8 string all day long and never ever attempt to check the validity of the string. Because, again, strlen() counts chars in a null-terminated char[]. It is giving you the right answer, you are asking it the wrong question. > Take your valid UTF-8 and change one of the characters to null, now it doesn't work in many circumstances with 'UTF-8' code. Which means it's not a valid UTF-8 decoder, but is instead treating the buffer as Modified UTF-8[1]. > that I could paste into a 'UNICODE' application Clipboards or pasteboards in many operating systems butcher character set when copying and pasting text. Generally, the clipboard cannot be trusted to do the right thing in every circumstance. On Windows, in particular, character set can get transposed to the system character set or something rather arbitrary when text is copied. > One was a UTF-8 string with BOM and has 0x00 inside :) (I call this string how to tell if it was written with C) > One was a UTF-8 string with a BOM :) Don't use the BOM[2] in UTF-8. It's recommended against. So really, your point is that some implementations are bad, and you have a bag of tricks for breaking implementations that don't handle all corner cases? That's pretty universal even in the non-Unicode world; there's bad implementations of everything. Windows is an especially bad implementation of most things Unicode. A valid decoder will, indeed, consistency-check an arbitrary string of bytes as UTF-8. The OP is correct, and your corner cases don't refute his point. [1]: http://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8 http://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8 [2]: http://en.wikipedia.org/wiki/UTF-8#Byte_order_mark http://en.wikipedia.org/wiki/UTF-8#Byte_order_mark
  - fleitz 15y ago
    A name like strlen suggests that it's designed to take the length of a string, if it was called count_null_ter_char_array then I'd tend to believe you. It's not character set agonistic, it's monotheistic at the shrine of ASCII, it's all over the coding style. Null is valid UTF-8, it just doesn't work with C 'strings'. I can get null out of a UTF-8 encoder with no problem. My point is that UTF-8 is nowhere near the panacea being described and if you have to touch the strings themselves that it's far better to use UTF-16 in the vast majority of cases. The only time you ever really want to use UTF-8 is if you're dealing with legacy codebases, it's a massive hack.

by pilif 15y ago

yuhong 15y ago

Yea, reminds me of DBCS. UTF-8 however don't use bytes below 0x80 as anything other than an ASCII character, unlike some DBCS encodings such as Shift-JIS.

_3u10 15y ago

"The one property I really love about UTF8 is that you get a free consistency check as not every arbitrary byte sequence is a valid UTF8 string." You don't get this at all using UTF-8. You only get it if you attempt to decode the string which even something like strlen doesn't do. Strlen will happily give you wrong answers about how many characters are in a UTF-8 string all day long and never ever attempt to check the validity of the string. Take your valid UTF-8 and change one of the characters to null, now it doesn't work in many circumstances with 'UTF-8' code. Also, should the free consistency check ever actually work you're in a bigger pickle as you now have to figure out whether the string is wrongly encoded UTF-8 or someone sent you extended ASCII. I did a lot of work with unicode apps. I used to have a series of about 5 strings that I could paste into a 'UNICODE' application and have it invariably break. One was an extended ASCII string that happend to be valid UTF-8 sans BOM :) One was a UTF-8 string with BOM and has 0x00 inside :) (I call this string how to tell if it was written with C) One was a UTF-8 string with a BOM :) One UTF-8 string with a some common latin characters, a couple japanese, and a character outside the BMP. Two UTF-16 strings in LE/BE with and sans BOM.

jsprinkles 15y ago
> You only get it if you attempt to decode the string which even something like strlen doesn't do. Because strlen() is a count of chars in a null-terminated char[], not a decoder. Ever. It's character set agnostic. > Strlen will happily give you wrong answers about how many characters are in a UTF-8 string all day long and never ever attempt to check the validity of the string. Because, again, strlen() counts chars in a null-terminated char[]. It is giving you the right answer, you are asking it the wrong question. > Take your valid UTF-8 and change one of the characters to null, now it doesn't work in many circumstances with 'UTF-8' code. Which means it's not a valid UTF-8 decoder, but is instead treating the buffer as Modified UTF-8[1]. > that I could paste into a 'UNICODE' application Clipboards or pasteboards in many operating systems butcher character set when copying and pasting text. Generally, the clipboard cannot be trusted to do the right thing in every circumstance. On Windows, in particular, character set can get transposed to the system character set or something rather arbitrary when text is copied. > One was a UTF-8 string with BOM and has 0x00 inside :) (I call this string how to tell if it was written with C) > One was a UTF-8 string with a BOM :) Don't use the BOM[2] in UTF-8. It's recommended against. So really, your point is that some implementations are bad, and you have a bag of tricks for breaking implementations that don't handle all corner cases? That's pretty universal even in the non-Unicode world; there's bad implementations of everything. Windows is an especially bad implementation of most things Unicode. A valid decoder will, indeed, consistency-check an arbitrary string of bytes as UTF-8. The OP is correct, and your corner cases don't refute his point. [1]: http://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8 http://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8 [2]: http://en.wikipedia.org/wiki/UTF-8#Byte_order_mark http://en.wikipedia.org/wiki/UTF-8#Byte_order_mark
- fleitz 15y ago
  A name like strlen suggests that it's designed to take the length of a string, if it was called count_null_ter_char_array then I'd tend to believe you. It's not character set agonistic, it's monotheistic at the shrine of ASCII, it's all over the coding style. Null is valid UTF-8, it just doesn't work with C 'strings'. I can get null out of a UTF-8 encoder with no problem. My point is that UTF-8 is nowhere near the panacea being described and if you have to touch the strings themselves that it's far better to use UTF-16 in the vast majority of cases. The only time you ever really want to use UTF-8 is if you're dealing with legacy codebases, it's a massive hack.