3 ms·
UTF-8 used to support up to 6 bytes to encode a unicode scalar value. Only later was the unicode range restricted such that a max of 4 bytes was required to enc
by Diggsey 5y ago
UTF-8 used to support up to 6 bytes to encode a unicode scalar value. Only later was the unicode range restricted such that a max of 4 bytes was required to encode any scalar value using UTF-8.
It looks like when UTF-8 was originally added to MySQL it still supported up to 6 bytes per "character", so the decision to restrict it was actually made by both MySQL and RFC 3629 - unfortunately they chose different limits :(
- tialaramex 5y agoWhat happened is Unicode committed forever to a range zero to U+10FFFF and RFC 3629 follows that. You don't need 5 or 6 byte UTF-8 to express this range, and so they're forbidden. So RFC 3629 isn't choosing arbitrarily, the choice in MySQL isn't entirely arbitrary either, it's just wrong (and should not have been named "utf8"). You could use the 3-byte UTF-8-like encoding MySQL has to do CESU-8 but that's not UTF-8. If MySQL wanted to do CESU-8 they could have called the type cesu8, giving away what's actually going on and necessitating a lot of extra work by MySQL. They did not choose to do that.
- Diggsey 5y agoRight, but my point was that the "U+10FFFF" limit hadn't been decided yet at the point where MySQL first added the "UTF-8" encoding, and it was added later, so it may not have been clear to them that UTF-8 would ever be a "fixed target" so to speak. In retrospect we can see that UTF-8 has been and will continue to be static, since it's become so universally adopted.