3 ms·

"An optimization was chosen to limit utf8 to 3 bytes, enough to handle almost all modern languages." I can't for the life of me imagine how, when discussing th

"An optimization was chosen to limit utf8 to 3 bytes, enough to handle almost all modern languages." I can't for the life of me imagine how, when discussing this in a room full of competent developers, everyone just went "great idea, lets do this". Just adds to the long list of questionable MySQL engineering decisions.

jeltz 8y ago
I am pretty sure it was not because of any optimization but that the real reason was to make sure a VARCHAR(255) would still fit within the maximum field size of InnoDB (768 bytes) at the time. Since then MySQL has added ROW_FORMAT=DYNAMIC to InnoDB which allows for larger values, but that probably required major changes to InnoDB.
- morgo 8y ago
  There are various buffers that were* not variable length in query execution (sorting, temp tables). It is not just about the on-disk format in this case. * Use MySQL 8.0 :-)
- MrBuddyCasino 8y ago
  There is usually a good reason for doing these kind of things. But then don't call it UTF8 encoding! It doesn't do that. It stores the most popular characters out of a larger address space. Call it UTF8-MB3 maybe from the start and let the user decide if having varchar(255) is worth the tradeoff. Thats the trouble with MySQL: doing silently the wrong thing, and thus violating the principle of least surprise. There is software where this might be ok, but a database is not that kind of software.
  - tinus_hn 8y ago
    In the case of MySQL (especially MySQL in 2003) bad design and silly behavior isn’t much of a surprise though :)
evanelias 8y ago
Sure, in hindsight, 3 bytes is an abomination to us today. That was almost certainly not apparent to developers at the time. MySQL added its three-byte utf8 support in mysql 4.1, first beta release in April 2003 [1]. Meanwhile the utf8 standard originally permitted up to 6 bytes per char, until officially changed in November 2003 [2]. Unicode support in web software -- and even programming languages -- was still rapidly evolving in 2003. The MySQL team probably just made a bad guess along the lines of "the standard allows 6 bytes but that has terrible perf and memory implications for us, and most people only use 3 bytes, so let's limit to that". While I've lost track of how many hundreds of times I've had to tell coworkers "use utf8mb4 and not utf8 in mysql", I don't begrudge the mysql dev team for not being clairvoyant in 2003! [1] https://lists.mysql.com/announce/161 https://lists.mysql.com/announce/161 [2] https://en.wikipedia.org/wiki/UTF-8#History https://en.wikipedia.org/wiki/UTF-8#History

jeltz 8y ago

I am pretty sure it was not because of any optimization but that the real reason was to make sure a VARCHAR(255) would still fit within the maximum field size of InnoDB (768 bytes) at the time. Since then MySQL has added ROW_FORMAT=DYNAMIC to InnoDB which allows for larger values, but that probably required major changes to InnoDB.

morgo 8y ago
There are various buffers that were* not variable length in query execution (sorting, temp tables). It is not just about the on-disk format in this case. * Use MySQL 8.0 :-)
MrBuddyCasino 8y ago
There is usually a good reason for doing these kind of things. But then don't call it UTF8 encoding! It doesn't do that. It stores the most popular characters out of a larger address space. Call it UTF8-MB3 maybe from the start and let the user decide if having varchar(255) is worth the tradeoff. Thats the trouble with MySQL: doing silently the wrong thing, and thus violating the principle of least surprise. There is software where this might be ok, but a database is not that kind of software.
- tinus_hn 8y ago
  In the case of MySQL (especially MySQL in 2003) bad design and silly behavior isn’t much of a surprise though :)

evanelias 8y ago

Sure, in hindsight, 3 bytes is an abomination to us today. That was almost certainly not apparent to developers at the time. MySQL added its three-byte utf8 support in mysql 4.1, first beta release in April 2003 [1]. Meanwhile the utf8 standard originally permitted up to 6 bytes per char, until officially changed in November 2003 [2]. Unicode support in web software -- and even programming languages -- was still rapidly evolving in 2003. The MySQL team probably just made a bad guess along the lines of "the standard allows 6 bytes but that has terrible perf and memory implications for us, and most people only use 3 bytes, so let's limit to that". While I've lost track of how many hundreds of times I've had to tell coworkers "use utf8mb4 and not utf8 in mysql", I don't begrudge the mysql dev team for not being clairvoyant in 2003! [1] https://lists.mysql.com/announce/161 https://lists.mysql.com/announce/161 [2] https://en.wikipedia.org/wiki/UTF-8#History https://en.wikipedia.org/wiki/UTF-8#History