9 ms·
Possibly my favourite thing about the rise of emoji is that they’re not Basic Multilingual Plane, and so stupid hacks like MySQL’s utf8 character set (I serious
by chrismorgan 8y ago
Possibly my favourite thing about the rise of emoji is that they’re not Basic Multilingual Plane, and so stupid hacks like MySQL’s utf8 character set (I seriously don’t know why anyone ever thought it was a good idea in 2002–2004) are now obviously insufficient for full Unicode support, and enough people care about them that adoption of this basic level of Unicode is driven, and so non-English speakers are inadvertently helped.
The next hurdle is to convince people that extended grapheme clusters are the smallest divisible unit, and that just like slicing at the byte level is insufficient if you’re doing UTF-8, slicing at the code point level is insufficient. Basically, strings should just be treated as opaque data in almost all places.
- philbarr 8y ago> not Basic Multilingual Plane, and so stupid hacks like the utf8 character set (I seriously don’t know why anyone ever thought it was a good idea in 2004) are now obviously insufficient for full Unicode support From [0]: UTF-8 was designed with a much larger limit of 231 (2,147,483,648) code points (32,768 planes), and can encode 221 (2,097,152) code points (32 planes) even if limited to 4 bytes What is wrong with UTF-8? [0]: https://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilingual_Plane https://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilin...
- cosarara 8y agoNot with utf-8 but the "utf8" mysql encoding.
- philbarr 8y agook - but parent referred to "utf8 character set". As if things weren't confusing enough already!
- chrismorgan 8y agoThey’re called character sets in SQL; see, for example, https://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8mb3.html https://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8... . I’ve clarified the original comment.
- philbarr 8y agoCan't reply to the sibling comment by Chris, but there isn't a "utf8 character set". There's just different encodings of a character set. Even after hunting around after going to the 404 you pointed me to, I still get this: utf8mb4: A UTF-8 encoding of the Unicode character set using one to four bytes per character. utf8mb3: A UTF-8 encoding of the Unicode character set using one to three bytes per character. So I don't think they are "different for SQL", whatever that means. Sorry - not trying to sound snarky but terminology is important and this is an area people easily get tripped up in. Especially me.
- chrismorgan 8y agoNot sure why the reply link is sometimes missing, but in my experience if you open just that comment (via the “N minutes ago” link) the textbox for replying is there and works. Perhaps the link is omitted to discourage rapid conversation? I fixed the link almost immediately—pesky inclusion of the dot in “.)” --- The SQL specification (I cite SQL-92) speaks of character sets (and uses the keyword CHARACTER SET). That outside of SQL, Unicode is the character set which happens to have multiple encodings, is immaterial; in SQL, each Unicode encoding is a CHARACTER SET—`CHARACTER SET utf8`, `CHARACTER SET utf8mb4`, whatever. This is the sense in which I used the term “character set”, the SQL sense. Before UCS-2 was proved insufficient, that encoding was actually called UNICODE, Wikipedia tells me. Concerning MySQL, utf8mb3 is an alias for utf8 which was introduced after utf8. I’m guessing it was introduced simultaneously with utf8mb4.
- philbarr 8y ago> utf8mb3 is an alias for utf8 which was introduced after utf8 I didn't know this - and although I don't use MySQL much these days it good to know, thank you. > in SQL, each Unicode encoding is a CHARACTER SET.... I see what you mean. That's pretty confusing... > but in my experience if you open just that comment (via the “N minutes ago” link) the textbox for replying is there and works It totally does!
- chrismorgan 8y ago
- deleted 8y ago[deleted]
- chc 8y agoIn case anyone is confused (since I know a lot of people see weird numbers and are just like, "Ugh, math hates me"), there's missing formatting in the above comment. It's supposed to read 2^31 and 2^21, not 231 and 221.
- deleted 8y ago[deleted]
- chrismorgan 8y agoWhile I’m contemplating why people might have thought utf8mb3 was wise in 2004, I’ll ask a similar question that I’d love to know the answer to, of events from my early childhood: why did UTF-16 gain popularity? I get UCS-2 existing, but once it became evident it wasn’t enough, why make UTF-16 and ruin Unicode with surrogate pairs (which still cause trouble today, leading to things like WTF-8 to make it possible for web browsers to abandon their UTF-16/UCS-2 mess in favour of almost-UTF-8)? UTF-8 was around by early 1993, while UTF-16 was developed several years later and seems fairly obviously the worst of both worlds—twice as large for normal text, ASCII-incompatible, and not even a fixed-width encoding, the only thing it almost had over UTF-8. The only thing it had going for it was that it was roughly compatible with UCS-2; but was UCS-2 that popular at the time? (Concerning UCS-2, hindsight says it should have been obvious 65,536 code points was going to be insufficient, but I’ll overlook that as an easy mistake to make; that part is better documented than the rise of UTF-16.) Possibly the most helpful description I have is from https://en.wikipedia.org/wiki/UTF-16#History https://en.wikipedia.org/wiki/UTF-16#History, but even that doesn’t explain why it became popular. Why did Java adopt it, Windows, browsers, &c.? Why?
- bhaak 8y agoI would disagree with the statement that UTF-16 gained popularity. UTF-16 is mostly used where it provided an easy upgrade path from the 16 bit restricted UCS-2 to enabling full Unicode coverage. Switching from UCS-2 to UTF-8 would have been a pain either because of dropping backward compatibility or ridiculous wrappers to get it working. Microsoft was one of the first companies to implement Unicode, so they didn't have much of a choice. Java could probably have switched to UTF-8 before 1.0 was released but I guess the supposed runtime overhead for not being able to access characters at indexes was feared too much. Of course, they lost that anyway with UTF-16.
- pwagland 8y agoJava 1.0 was made back in the days when "65K characters would be enough for anyone!" hence they chose UCS-2. Java only supported UTF-16 as of Java 5. I suspect that if they could turn back time, they would have chosen UTF-8 from the get-go, but, as you say, that was a battle lost long ago.
- deleted 8y ago[deleted]
- amelius 8y ago> Possibly my favourite thing about the rise of emoji is that they’re not Basic Multilingual Plane When can we have user-defined glyphs/emoji, where SVG is embedded in the unicode string?
- chrismorgan 8y agoYou’d just define two code points that switches into and out of SVG mode. (The second is only necessary for performance, so you can scan quickly without needing any sort of XML parser!)
- ithkuil 8y agoWhat if nested <text> element also containers an inline emosvg? :-)
- chrismorgan 8y agoConcerning the SVG table in OpenType fonts, https://docs.microsoft.com/en-au/typography/opentype/spec/svg#security-considerations-and-other-glyph-rendering-restrictions https://docs.microsoft.com/en-au/typography/opentype/spec/sv... declares: > any SVG <text> and <foreignObject> elements within a glyph description must be ignored and not rendered The same limitations would be applied to this hypothetical SVG mode.
- bmn__ 8y agoYou can have that now, but it doesn't work the way you think. The SVG is embedded in the font, not the string. Demo: https://eosrei.github.io/emojione-color-font/full-demo.html https://eosrei.github.io/emojione-color-font/full-demo.html To create user-defined characters, you can occupy any code point in the reserved areas. https://en.wikipedia.org/wiki/Private_Use_Areas https://en.wikipedia.org/wiki/Private_Use_Areas
- amelius 8y agoThat's nice, but I meant something else. I was thinking: what if users of Whatsapp and other chat services could define their own emoji, as they type, and share them with other users? If a friend sends you a new emoji, you could simply copy it and use it in messages of your own. This functionality doesn't work with fixed codepoints. Of course there's a million way to do this (even with today's Unicode standard), but it would be nice if this kind of functionality would be part of a standard, perhaps even at the level of Unicode.
- paidleaf 8y ago> and so stupid hacks like MySQL’s utf8 character set (I seriously don’t know why anyone ever thought it was a good idea in 2002–2004) are now obviously insufficient for full Unicode support If I had to guess, they did it for space and performance reasons so that their benchmark results are a tiny bit better than other RDBMs. They must've figured that their limited utf8 encoding was enough. And they reasoned the immediate benefits on benchmark tests outweighed future considerations. I can't imagine any other reason for not implementing the utf-8 encoding completely and covering the unicode character set in its entirety.
- morgo 8y agoCorrect.
- deleted 8y ago[deleted]
- OnlyRepliesToBS 8y agoIt's a big fucking mistake is what it is.
- udp 8y agoThere’s nothing wrong with slicing at the byte level if you’re slicing with a delimiter like \n or “. UTF-8 sets high bits so continuation bytes never look like ASCII characters.
- freeone3000 8y agoCombined emoji, such as 🇺🇸, are great for this as well! Things like also help - separating them out, even on spaces, breaks the emoji. We are really making progress towards proper processing.