3 ms·

"those bytes inherently have a different meaning" Are you sure? Utf-8 is a variable-width encoding so it should just work, right?

evanelias 5y ago
Yes, you're correct that utf8mb3 is strictly a subset of utf8mb4, and the binary value of any valid utf8mb3 character is identical to its utf8mb4 representation. But I'd argue that's irrelevant. I meant my statement generically: the meaning of the bytes stored in a column value depend entirely on the column type, as does the range of values that may be stored or converted. If a column's type is, say, VARCHAR(100) CHARACTER SET utf8mb3, this means the column can store up to 100 utf8 characters and each character has a maximum length of 3 bytes. It is simply not legal to store 4-byte characters in this data type, just as it isn't legal to store 101 characters in it. The proposal in this subthread seems to be, instead of introducing utf8mb4, why didn't MySQL allow "utf8" (utf8mb3) columns store 4-byte characters, just without affecting the maximum number of bytes that can be stored in the column? So, a VARCHAR(100) CHARACTER SET utf8 might only be able to store max 75 characters if they're all emoji. Additionally, it would mean that the range of values allowed in a column would depend on the DB server version, and would change upon upgrading. This just seems completely counterintuitive and wrong to me, relative to how all other column type definitions work. The modern progression of MySQL has been to remove these obvious-WTF problems, not introduce more of them :) The proposal was described as "seems less bad than the alternative of silently corrupting data", but that isn't the alternative (silent data corruption doesn't occur with strict mode which is enabled by default), so this comparison doesn't make sense.
kevin_thibedeau 5y ago
Everything that decodes that utf-8 has to support 21-bit codepoints. If they used a uint16_t anywhere in the processing chain then it's restricted to the BMP.

evanelias 5y ago

Yes, you're correct that utf8mb3 is strictly a subset of utf8mb4, and the binary value of any valid utf8mb3 character is identical to its utf8mb4 representation. But I'd argue that's irrelevant. I meant my statement generically: the meaning of the bytes stored in a column value depend entirely on the column type, as does the range of values that may be stored or converted. If a column's type is, say, VARCHAR(100) CHARACTER SET utf8mb3, this means the column can store up to 100 utf8 characters and each character has a maximum length of 3 bytes. It is simply not legal to store 4-byte characters in this data type, just as it isn't legal to store 101 characters in it. The proposal in this subthread seems to be, instead of introducing utf8mb4, why didn't MySQL allow "utf8" (utf8mb3) columns store 4-byte characters, just without affecting the maximum number of bytes that can be stored in the column? So, a VARCHAR(100) CHARACTER SET utf8 might only be able to store max 75 characters if they're all emoji. Additionally, it would mean that the range of values allowed in a column would depend on the DB server version, and would change upon upgrading. This just seems completely counterintuitive and wrong to me, relative to how all other column type definitions work. The modern progression of MySQL has been to remove these obvious-WTF problems, not introduce more of them :) The proposal was described as "seems less bad than the alternative of silently corrupting data", but that isn't the alternative (silent data corruption doesn't occur with strict mode which is enabled by default), so this comparison doesn't make sense.

kevin_thibedeau 5y ago

Everything that decodes that utf-8 has to support 21-bit codepoints. If they used a uint16_t anywhere in the processing chain then it's restricted to the BMP.