3 ms·
> While I’m contemplating why people might have thought utf8mb3 was wise in 2004 Because this is 2004 MySQL, they weren't exactly known for technical acumen an
by masklinn 8y ago
> While I’m contemplating why people might have thought utf8mb3 was wise in 2004
Because this is 2004 MySQL, they weren't exactly known for technical acumen and forward thinking. The root of their decision is that behind the scenes "UTF8MB4" stores codepoints in fixed-size areas.
And MySQL could not index columns larger than 767 bytes.
Guess what doesn't fit in 767 bytes? VARCHAR(255) in UTF8MB4. Anything above VARCHAR(191) would not be indexable.
Meanwhile VARCHAR(255) with 3 bytes per codepoint fits just right.
So rather than fix their crap, they just decided to limit their "UTF8" to the BMP and voilà no problem.
> The only thing it had going for it was that it was roughly compatible with UCS-2; but was UCS-2 that popular at the time?
The foundations of major systems (e.g. WinNT or Java) had long been laid out (in fact WinNT was released in 1993), and while UTF8 did exist there were few use cases for going beyond the BMP; and people had (and by and large still have) the misguided view that O(1) indexation in a string is a valuable property, and so their POV was that their choice was UCS2+surrogates (which few people would ever encounter) or UCS4.
> Possibly the most helpful description I have is from https://en.wikipedia.org/wiki/UTF-16#History https://en.wikipedia.org/wiki/UTF-16#History, but even that doesn’t explain why it became popular. Why did Java adopt it, Windows, browsers, &c.? Why?
What they adopted was UCS2, this was "backronymed" to a bastard (non-transparent, you'd see every code unit) UTF16 later on. Java didn't even support getting codepoints out of strings until Java 5 in the mid aughts: https://docs.oracle.com/javase/8/docs/api/java/lang/String.html#codePointAt-int- https://docs.oracle.com/javase/8/docs/api/java/lang/String.h...
I'm pretty sure most string processing in Java and C# remains unaware of surrogate pairs and can corrupt proper UTF16 data rather interestingly. That is most definitely an extant issue in Javascript (not that properly handling that is even remotely sufficient to do actual unicode-aware string processing).
- morgo 8y ago> Guess what doesn't fit in 767 bytes? VARCHAR(255) in UTF8MB4. Anything above VARCHAR(191) would not be indexable. The utf8mb3 optimization was made before InnoDB was the default storage engine (2010).
- sulam 8y agoIt was widely used before it was the default.
- morgo 8y agoCertainly, but let’s not rewrite history here. What innodb could/couldn’t do was not the motivation.
- masklinn 8y agoWhy not? Even as not the default, it was an official bundled storage engine, and the cutoff just happens to occur right at its boundaries, that doesn't sound like a coincidence.
- liveoneggs 8y agoI like how your write this as if I'm not still running into these index column size issues today ;)
- deleted 8y ago[deleted]