4 ms·

While searching yesterday, I recall the mysql documentation did specify this this quirk. As I gather, this change happened prior to or at the change to 4 char

by petee 5y ago

While searching yesterday, I recall the mysql documentation did specify this this quirk. As I gather, this change happened prior to or at the change to 4 char max utf; was supplemental plane support required at that time? 3 char is still utf8, just not fully supported, but I really do not know. Not to defend, just saying things were new and fluid back then, though it should have been fixed properly alot sooner.

tialaramex 5y ago
In Modern MySQL the documentation does - if you look - tell you that utf8mb4 exists to actually do UTF-8 and if you think to check it mentions that their utf8 is not in fact UTF-8 but only "utf8mb3". But fifteen years ago most of that documentation doesn't exist, and utf8mb4 doesn't exist, and if you either read the MySQL documentation carefully and realise "utf8" isn't UTF-8, and only encodes the Basic Multilingual Plane, you are stuck anyway, MySQL doesn't have a sensible way forward for you. I would argue that by 2002 when this bug (let's call it what it is) was introduced, the writing was on the wall, UCS-2 is dead, which means the "it's just easier" justification for choosing 16-bit is also dead, so UTF-8 is the natural choice for software that isn't meanwhile committed to a 16-bit encoding anyway (as Java and Windows are). That makes shipping UTF-8 a good idea, but it makes choosing utf8mb3 a really weird idea that already seems certain to cause pain.
- petee 5y ago
  Could be. And its weird especially when supporting all 4 bytes would be easy (so it seems). For fun I did dig around though, and found that even in the earliest documentation they clearly state only supporting 3 bytes, though not why. It says this in 2005 which only justifies not supporting 5/6 bytes, but not four. So confusing: "RFC 3629 describes encoding sequences that take from one to four bytes. Currently, MySQL UTF8 support does not include four-byte sequences. (An older standard for UTF8 encoding is given by RFC 2279, which describes UTF8 sequences that take from one to six bytes. RFC 3629 renders RFC 2279 obsolete; for this reason, sequences with five and six bytes are no longer used.)" ...Wha? (2003/2005, they changed urls) - https://web.archive.org/web/20031211033604/http://www.mysql.com/doc/en/Charset-Unicode.html https://web.archive.org/web/20031211033604/http://www.mysql.... - https://web.archive.org/web/20050226012911/http://dev.mysql.com/doc/mysql/en/charset-unicode.html https://web.archive.org/web/20050226012911/http://dev.mysql.... I personally think anything critical that was worth storing in a new character format still should have been properly tested and documentation consulted, that is on the end-user. Anything not critical where you wouldn't expect to care...still should have tested, but yeah that sucks.

by petee 5y ago

tialaramex 5y ago

In Modern MySQL the documentation does - if you look - tell you that utf8mb4 exists to actually do UTF-8 and if you think to check it mentions that their utf8 is not in fact UTF-8 but only "utf8mb3". But fifteen years ago most of that documentation doesn't exist, and utf8mb4 doesn't exist, and if you either read the MySQL documentation carefully and realise "utf8" isn't UTF-8, and only encodes the Basic Multilingual Plane, you are stuck anyway, MySQL doesn't have a sensible way forward for you. I would argue that by 2002 when this bug (let's call it what it is) was introduced, the writing was on the wall, UCS-2 is dead, which means the "it's just easier" justification for choosing 16-bit is also dead, so UTF-8 is the natural choice for software that isn't meanwhile committed to a 16-bit encoding anyway (as Java and Windows are). That makes shipping UTF-8 a good idea, but it makes choosing utf8mb3 a really weird idea that already seems certain to cause pain.

petee 5y ago
Could be. And its weird especially when supporting all 4 bytes would be easy (so it seems). For fun I did dig around though, and found that even in the earliest documentation they clearly state only supporting 3 bytes, though not why. It says this in 2005 which only justifies not supporting 5/6 bytes, but not four. So confusing: "RFC 3629 describes encoding sequences that take from one to four bytes. Currently, MySQL UTF8 support does not include four-byte sequences. (An older standard for UTF8 encoding is given by RFC 2279, which describes UTF8 sequences that take from one to six bytes. RFC 3629 renders RFC 2279 obsolete; for this reason, sequences with five and six bytes are no longer used.)" ...Wha? (2003/2005, they changed urls) - https://web.archive.org/web/20031211033604/http://www.mysql.com/doc/en/Charset-Unicode.html https://web.archive.org/web/20031211033604/http://www.mysql.... - https://web.archive.org/web/20050226012911/http://dev.mysql.com/doc/mysql/en/charset-unicode.html https://web.archive.org/web/20050226012911/http://dev.mysql.... I personally think anything critical that was worth storing in a new character format still should have been properly tested and documentation consulted, that is on the end-user. Anything not critical where you wouldn't expect to care...still should have tested, but yeah that sucks.