6 ms·

The 2002 diff got me interested in the context of why that change was made, which lead me to this 2006 bug which I think might shed light on why '3' was chosen:

by petee 5y ago

The 2002 diff got me interested in the context of why that change was made, which lead me to this 2006 bug which I think might shed light on why '3' was chosen: "The UTF-8 encoding is a variable byte length encoding. The ASCII subrange of characters is one byte long, alphabetic writings 2 bytes long, Far Eastern writings 3 bytes long per character (and some rare historical writings are mapped using 4 bytes)." https://bugs.mysql.com/bug.php?id=18560#c61151 https://bugs.mysql.com/bug.php?id=18560#c61151 UTF-8 is still something I'm learning the details about, but it looks like back in 2002 they simply didn't see the need for the supplemental plane characters, ie, playing cards, arrows, or ancient Greek numbers. It might be short sighted but we shouldn't judge so far in the future, recalling that emoji wasn't even in Unicode until 2010.

deleted 5y ago
[deleted]
Diggsey 5y ago
UTF-8 used to support up to 6 bytes to encode a unicode scalar value. Only later was the unicode range restricted such that a max of 4 bytes was required to encode any scalar value using UTF-8. It looks like when UTF-8 was originally added to MySQL it still supported up to 6 bytes per "character", so the decision to restrict it was actually made by both MySQL and RFC 3629 - unfortunately they chose different limits :(
- tialaramex 5y ago
  What happened is Unicode committed forever to a range zero to U+10FFFF and RFC 3629 follows that. You don't need 5 or 6 byte UTF-8 to express this range, and so they're forbidden. So RFC 3629 isn't choosing arbitrarily, the choice in MySQL isn't entirely arbitrary either, it's just wrong (and should not have been named "utf8"). You could use the 3-byte UTF-8-like encoding MySQL has to do CESU-8 but that's not UTF-8. If MySQL wanted to do CESU-8 they could have called the type cesu8, giving away what's actually going on and necessitating a lot of extra work by MySQL. They did not choose to do that.
  - Diggsey 5y ago
    Right, but my point was that the "U+10FFFF" limit hadn't been decided yet at the point where MySQL first added the "UTF-8" encoding, and it was added later, so it may not have been clear to them that UTF-8 would ever be a "fixed target" so to speak. In retrospect we can see that UTF-8 has been and will continue to be static, since it's become so universally adopted.
tialaramex 5y ago
If it was called just "string" or something and then it quietly explained somewhere that it's actually only the BMP and not all of Unicode that's not so surprising, but they deliberately called it UTF-8 even though it just isn't. Imagine if you bought a USB mouse and then you discovered, oh, they didn't bother implementing USB, this is just a PS/2 mouse with the USB connector. Works fine so long as you use the supplied PS/2 "adaptor" but it isn't actually USB even though it says it is on the box. That mouse is defective. MySQL is defective as supplied. Which won't come as a surprise to many experienced MySQL developers. I actually rather like MySQL, but it's like bash or Excel, it is a good tool for quick estimates or prototyping, and a poor tool for actual engineering. For the cost of all the work needed to get reliable high quality results with MySQL you could have used a different database where the baseline effort to get it working was higher but what you get is more solid, and "utf8mb4" is an example of that extra work in MySQL.
- petee 5y ago
  While searching yesterday, I recall the mysql documentation did specify this this quirk. As I gather, this change happened prior to or at the change to 4 char max utf; was supplemental plane support required at that time? 3 char is still utf8, just not fully supported, but I really do not know. Not to defend, just saying things were new and fluid back then, though it should have been fixed properly alot sooner.
  - tialaramex 5y ago
    In Modern MySQL the documentation does - if you look - tell you that utf8mb4 exists to actually do UTF-8 and if you think to check it mentions that their utf8 is not in fact UTF-8 but only "utf8mb3". But fifteen years ago most of that documentation doesn't exist, and utf8mb4 doesn't exist, and if you either read the MySQL documentation carefully and realise "utf8" isn't UTF-8, and only encodes the Basic Multilingual Plane, you are stuck anyway, MySQL doesn't have a sensible way forward for you. I would argue that by 2002 when this bug (let's call it what it is) was introduced, the writing was on the wall, UCS-2 is dead, which means the "it's just easier" justification for choosing 16-bit is also dead, so UTF-8 is the natural choice for software that isn't meanwhile committed to a 16-bit encoding anyway (as Java and Windows are). That makes shipping UTF-8 a good idea, but it makes choosing utf8mb3 a really weird idea that already seems certain to cause pain.
chmod600 5y ago
What we can judge harshly is that it accepted the 4-byte sequences and failed silently, corrupting data. A sane system would reject the 4-byte sequences so you could try again without the emojis. But if users got errors, they might investigate, and it would slow down the rapid growth mysql was aiming for. So they just decided not to throw errors.
- petee 5y ago
  I agree, although the change was prior to 4-byte utf (2002 vs 2003) the same issue would have manifested with 6 byte sequences. Very curious. It would be really interesting to see a postmortem by mysql if ever possible, because the more I think about, 3 is such an odd choice (pun), and people usually do most things for a reason haha, its either lost to time, or a brainfart off-by-one.

by petee 5y ago

deleted 5y ago

[deleted]

Diggsey 5y ago

UTF-8 used to support up to 6 bytes to encode a unicode scalar value. Only later was the unicode range restricted such that a max of 4 bytes was required to encode any scalar value using UTF-8. It looks like when UTF-8 was originally added to MySQL it still supported up to 6 bytes per "character", so the decision to restrict it was actually made by both MySQL and RFC 3629 - unfortunately they chose different limits :(

tialaramex 5y ago
What happened is Unicode committed forever to a range zero to U+10FFFF and RFC 3629 follows that. You don't need 5 or 6 byte UTF-8 to express this range, and so they're forbidden. So RFC 3629 isn't choosing arbitrarily, the choice in MySQL isn't entirely arbitrary either, it's just wrong (and should not have been named "utf8"). You could use the 3-byte UTF-8-like encoding MySQL has to do CESU-8 but that's not UTF-8. If MySQL wanted to do CESU-8 they could have called the type cesu8, giving away what's actually going on and necessitating a lot of extra work by MySQL. They did not choose to do that.
- Diggsey 5y ago
  Right, but my point was that the "U+10FFFF" limit hadn't been decided yet at the point where MySQL first added the "UTF-8" encoding, and it was added later, so it may not have been clear to them that UTF-8 would ever be a "fixed target" so to speak. In retrospect we can see that UTF-8 has been and will continue to be static, since it's become so universally adopted.

tialaramex 5y ago

If it was called just "string" or something and then it quietly explained somewhere that it's actually only the BMP and not all of Unicode that's not so surprising, but they deliberately called it UTF-8 even though it just isn't. Imagine if you bought a USB mouse and then you discovered, oh, they didn't bother implementing USB, this is just a PS/2 mouse with the USB connector. Works fine so long as you use the supplied PS/2 "adaptor" but it isn't actually USB even though it says it is on the box. That mouse is defective. MySQL is defective as supplied. Which won't come as a surprise to many experienced MySQL developers. I actually rather like MySQL, but it's like bash or Excel, it is a good tool for quick estimates or prototyping, and a poor tool for actual engineering. For the cost of all the work needed to get reliable high quality results with MySQL you could have used a different database where the baseline effort to get it working was higher but what you get is more solid, and "utf8mb4" is an example of that extra work in MySQL.

petee 5y ago
While searching yesterday, I recall the mysql documentation did specify this this quirk. As I gather, this change happened prior to or at the change to 4 char max utf; was supplemental plane support required at that time? 3 char is still utf8, just not fully supported, but I really do not know. Not to defend, just saying things were new and fluid back then, though it should have been fixed properly alot sooner.
- tialaramex 5y ago
  In Modern MySQL the documentation does - if you look - tell you that utf8mb4 exists to actually do UTF-8 and if you think to check it mentions that their utf8 is not in fact UTF-8 but only "utf8mb3". But fifteen years ago most of that documentation doesn't exist, and utf8mb4 doesn't exist, and if you either read the MySQL documentation carefully and realise "utf8" isn't UTF-8, and only encodes the Basic Multilingual Plane, you are stuck anyway, MySQL doesn't have a sensible way forward for you. I would argue that by 2002 when this bug (let's call it what it is) was introduced, the writing was on the wall, UCS-2 is dead, which means the "it's just easier" justification for choosing 16-bit is also dead, so UTF-8 is the natural choice for software that isn't meanwhile committed to a 16-bit encoding anyway (as Java and Windows are). That makes shipping UTF-8 a good idea, but it makes choosing utf8mb3 a really weird idea that already seems certain to cause pain.

chmod600 5y ago

What we can judge harshly is that it accepted the 4-byte sequences and failed silently, corrupting data. A sane system would reject the 4-byte sequences so you could try again without the emojis. But if users got errors, they might investigate, and it would slow down the rapid growth mysql was aiming for. So they just decided not to throw errors.

petee 5y ago
I agree, although the change was prior to 4-byte utf (2002 vs 2003) the same issue would have manifested with 6 byte sequences. Very curious. It would be really interesting to see a postmortem by mysql if ever possible, because the more I think about, 3 is such an odd choice (pun), and people usually do most things for a reason haha, its either lost to time, or a brainfart off-by-one.