9 ms·
This is why UTF-8 is great. If it works for any Unicode character it will work for them all. Surrogate pairs are rare enough that they are poorly tested. With U
by oofabz 14y ago
This is why UTF-8 is great. If it works for any Unicode character it will work for them all. Surrogate pairs are rare enough that they are poorly tested. With UTF-8, if there are issues with multi-byte characters, they are obvious enough to get fixed.
UTF-16 is not a very good encoding. It only exists for legacy reasons. It has the same major drawback as UTF-8 (variable-length encoding) but none of the benefits (ASCII compatibility, size efficient).
- pixelcort 14y agoThe problem with UTF-8 is that lots of tools have 3 byte limits, and characters like Emoji take up 4 bytes in UTF-8.
- Tsagadai 14y agoFile bugs with those tools. These sort of issues should have been sorted years ago and any program that can't do 3+ byte character encodings should be named and shamed.
- potatolicious 14y agoHonest question: but isn't that just a broken implementation (and a very obvious brokenness at that)? It seems to me there's a big difference between someone not coding to the standard, and the standard making your taks impossible.
- dsl 14y agoIf most implementations are broken, it becomes the standard.
- codeka 14y agoThe same could be said of UTF-16 implementations that don't support surrogate pairs.
- tlrobinson 14y agoBut that's not true for UCS-2, which can't represent certain code points. On the other hand, UTF-8 is ASCII compatible and more efficient for text that's primarily ASCII.
- thristian 14y agoI don't think it's specifically a 3-byte limit, I think it's just that lots of tools decode UTF-8 into UCS-2 internally instead of UTF-16.
- jrabone 14y agoThat is quite clearly broken, and any tool that does so should be fixed or dumped. This is not new, and Marcus Kuhn had made UTF8 test resources available for years at http://www.cl.cam.ac.uk/~mgk25/unicode.html http://www.cl.cam.ac.uk/~mgk25/unicode.html
- mef 14y agoI've found this sub-page super useful over the years for testing http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt
- thristian 14y agoI believe that MySQL is one such tool. In recent versions you can work around it by asking for the encoding "utf8mb4" instead of "utf8", but I think they have to be quite recent. So yes, another way in which MySQL is quite clearly broken.
- derleth 14y agoAnd the problem with UTF-16 is that a lot of applications can't handle surrogate pairs, except a lot of Emoji are above the BMP, aren't they? So why is this a bigger deal for UTF-8 than UTF-16?
- masklinn 14y ago> except a lot of Emoji are above the BMP, aren't they? All of the Unicode 6.0 emoji are.
- eps 14y agoWhich tools? Honest question, as the three byte limit seems rather arbitrary and no more logical than, say, a four byte one.
- jrabone 14y agoIt is totally arbitrary - there's no reason you can't have degenerate 6-byte encodings, and compliant decoders should cope with them. See Marcus Kuhn's excellent UTF-8 decoder torture test page linked elsewhere in this thread.
- mikeash 14y agoActually, a truly compliant UTF-8 decoder should reject all degenerate forms, because they can be used to bypass validation checks, and the spec does not allow such forms. A four-byte UTF-8 sequence is sufficient to represent any Unicode code point (and then some), so no compliant decoder should accept any sequences of 5 or 6 bytes. Note that Kuhn's torture test page deliberately includes a lot of invalid sequences in order to make sure that they're gracefully handled. Section 4 is dedicated to degenerate encodings like this.
- masklinn 14y agoIt's not 3 bytes so much as 16 bits, aka Unicode 1.0 limits. Which turn into 3 bytes in UTF-8.
- pjscott 14y agoHow many tools have 3-byte limits on UTF-8? The only one I can think of right now is MySQL. (The workaround is to specify the utf8mb4 character set. This is MySQL's cryptic internal name for "actually doing UTF-8 correctly.")
- jrabone 14y agoMySQL is one of the worst offenders for broken Unicode and collation problems arising therein. Neither it nor JavaScript deserve consideration for problems that need robust Unicode handling.
- mikeash 14y agoI actually switched my (low traffic, low performance needs) blog comments database from MySQL to SQLite purely because I could not make MySQL and Unicode get along. All I needed was for it to accept and then regurgitate UTF-8 and it couldn't even handle that. I'm sure it can be done, but none of the incantations I tried made it work, and it was ultimately easier for me to switch databases.
- pjscott 14y agoAs an ugly last resort, you could store Unicode as UTF-8 in BLOB fields. MySQL is pretty good about storing binary data. (I dread the day that I'll have to do something more advanced with Unicode in MySQL than just storing it.)
- mikeash 14y agoI no longer recall whether I tried that and failed, or didn't get that far. Seems like a semi-reasonable approach if you don't need the database to be able to understand the contents of that column. But on the other hand, SQLite is working great for my needs too.
- deleted 14y ago[deleted]
- deleted 14y ago
- deleted 14y ago[deleted]
- est 14y agoUCS2 is better than UTF8 internally because it counts unicode characters faster than UTF8. Every character is juest 2 bytes, instead of 1, 2, 3 or even 4 bytes. In python: len(u'汉字') == 2 len( '汉字') == 4 # or maybe 6, it varies based on console encoding and CPython options len(u'汉字'.encode('utf8')) == 6
- dietrichepp 14y agoWhy do you want to count Unicode characters? Why do you care if it is fast to do so? Why would you ever need to use character-based string indexing? UTF-16 solves problems that don't exist. (Honestly, I would love it if someone could explain what the purpose of counting characters is, because I don't know why you'd ever do that, except when you're posting to Twitter.)
- deleted 14y ago[deleted]
- pjscott 14y agoIf you make an API, please don't do this. If you have to include indexes into a Unicode string, then make them indexes into a binary string in a known encoding. This works equally well for all encodings, and won't tempt people to do something as preposterous as using UCS-2.
- mikeash 14y agoSuch an API could easily indicate bytes in a UTF-8 string instead.
- marshray 14y agoYou will never define an unambiguous way to index into a Unicode string by "characters" with less than 10 pages of specification. When two sides of the API interpret this contract differently, a security vulnerability is a very possible outcome. Use unambiguous delimiters, JSON, XML, or even byte offsets into normalized UTF-8 instead. But don't do this please.
- 14y ago
- notJim 14y agoThis comment is somewhat misleading. The issue at hand is orthogonal to any of the benefits of UTF-8 over UTF-16 (which are real, UTF-8 is great, you should use it.) 4-byte characters in UTF-8 are just as rare as surrogate pairs are just as rare in UTF-16, because they both are used to represent non-BMP characters. As a result, there is software that handles 3-byte characters (i.e., a huge percentage of what you'll ever see), but doesn't handle 4-byte characters. MySQL is a high-profile example of software which, until recently, had this problem: http://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8mb4.html http://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8m....
- snogglethorpe 14y agoThe problem is that handling 1 unit is very different from 2+ units, in terms of coding patterns, whereas 3 is not so different from 4+. In the latter case there's already probably a loop to handle multiple unit characters, which will in many cases work without change for longer sequences (and if not, probably the code probably requires very little change to do so). So whereas it's rather common for programs to mis-handle multiple-unit UTF-16 characters, it seems much less likely that programs will mis-handle 4+ unit UTF-8 characters.