5 ms·
A good summary, but for one imortant detail: In UTF-16, some code points (laying on the so-called "astral planes", ie not on the "basic multilingual plane") tak
by pygy_ 13y ago
A good summary, but for one imortant detail: In UTF-16, some code points (laying on the so-called "astral planes", ie not on the "basic multilingual plane") take 32 bits.
The Emoji, for example, lie on the first higher plane: 🍒🎄🐰🚴. Firefox and Safari display them properly, Chrome doesn't, no idea for IE and Opera.
UCS-2 is a strict 16-bit encoding (a subset of UTF-16), and it cannot represent all characters.
It is the encoding used by JavaScript, which can be problematic when double width characters are used. For example, `"🐙🐚🐛🐜🐝🐞🐟".length` is 14 even though there are only seven characters, and you could slice that string in the middle of a character.
- brazzy 13y agoIt should be noted that at the time JavaScript was developed, there was only the basic multilingual plane - the extension was only introduced with Unicode 2.0 in 1996.
- shousper 13y agoI wonder how hard it'd be to get JavaScript/ECMAScript onto a better encoding.. Do we actually have a "better" encoding?
- Scorponok 13y agoDepends what you mean by "better". UTF-8 generally ends up using fewer bytes to represent the same string than UTF-16, unless you're using certain characters a lot (e.g. for asian languages), so it's a candidate, but it's not like you could just flip a switch and make all javascript use UTF-8.
- oofabz 13y agoI think the size issue is a red herring. UTF-8 wins some, UTF-16 wins others, but either encoding is acceptable. There is no clear winner here so we should look at other properties. UTF-8 is more reliable, because mishandling variable-length characters is more obvious. In UTF-16 it's easy to write something that works with the BMP and call it good enough. Even worse, you may not even know it fails above the BMP, because those characters are so rare you might never test with them. But in UTF-8, if you screw up multi-byte characters, any non-ASCII character will trigger the bug, and you will fix your code more quickly. Also, UTF-8 does not suffer from endianness issues like UTF-16 does. Few people use the BOM and no one likes it. And most importantly, UTF-8 is compatible with ASCII.
- wrhs 13y agoBack in the real world though, UTF-16 almost never "wins" size comparisons in practice. For example, one would think that UTF-16 would be "better" than UTF-8 for pure Thai documents (2 bytes vs 3 bytes per Thai character) but in reality such a comparison is irrelevant, since the Thai government have decided that both are unacceptable in most cases and instead mandate the use of TIS-620 (1 byte per ASCII or Thai character). It's a similar story in almost every other case where UTF-16 may seem like the best choice for size purposes. There's usually a specialist or national standard encoding in place, with much more traction (and often required by law), that is smaller and doesn't share any of the heinous issues suffered by UTF-16. tl;dr: I don't think you made the point strongly enough. UTF-16 is an absolute joke and will only stay around due to Microsoft's stubbornness and lack of foresight. If you voluntarily use UTF-16 in 2014, you are a moron.
- lucian1900 13y agoThere is absolutely no situation in which UTF-16 wins over UTF-8, because of the surrogate pairs required. That makes both encodings variable length. UTF-32 is probably what you're thinking of.
- oofabz 13y agoI know that both encodings are variable-length. That is the issue I am trying to address. My point is that in UTF-16 it's too easy to ignore surrogate pairs. Lots of UTF-16 software fails to handle variable-length characters because they are so rare. But in UTF-8 you can't ignore multi-byte characters without obvious bugs. These bugs are noticed and fixed more quickly than UTF-16 surrogate pair bugs. This makes UTF-8 more reliable. I am not sure why you think I am advocating UTF-16. I said almost nothing good about it.
- millstone 13y agoBugs in UTF-8 handling of multibyte sequences need not be obvious. Google "CAPEC-80." UTF-16 has an advantage in that there's fewer failure modes, and fewer ways for a string to be invalid. edit: As for surrogate pairs, this is an issue, but I think it's overstated. A naïve program may accidentally split a UTF-16 surrogate pair, but that same program is just as liable to accidentally split a decomposed character sequence in UTF-8. You have to deal with those issues regardless of encoding.
- madeofpalk 13y agoFrom my undestanding, the Chrome doesn't display emoji because of poor unicode handling, but its a difference in the (font?) renderer used. See http://apple.stackexchange.com/questions/41228/why-do-emoji-like-appear-when-i-use-safari-but-not-chrome http://apple.stackexchange.com/questions/41228/why-do-emoji-... and https://code.google.com/p/chromium/issues/detail?id=90177 https://code.google.com/p/chromium/issues/detail?id=90177
- eurleif 13y agoThis is a bit pedantic, but JS implementations don't necessarily use UCS-2 as the internal encoding. The issue is that the spec requires characters to be exposed to programs as 16-bit values. See here: http://mathiasbynens.be/notes/javascript-encoding http://mathiasbynens.be/notes/javascript-encoding
- golergka 13y agoEmojis are even more fun than that. Some of them take two unicode characters.
- wrhs 13y agoI think you mean "two UTF-16 codepoints". "Unicode" and UTF-16 are not interchangeable terms. Nor are "character" and codepoint.
- wrhs 13y agoUmm, this is like Unicode 101. Mentioning it here is just patronizing.
- hobs 13y ago🚴 isnt displayed correctly in my Firefox, though all the other characters you mention are. FF 26 on Win 7.