5 ms·

WTF business do emojis have in Unicode? The BMP is all there ever should have been. Standardize the actual writing systems of the world, so everyone can write i

WTF business do emojis have in Unicode? The BMP is all there ever should have been. Standardize the actual writing systems of the world, so everyone can write in their language. And once that is done, the standard doesn't need to change for a hundred years. What we need now is a standardized, sane subset of Unicode that implementations can support while rejecting the insane scope creep that got added on top of that. I guess the BMP is a good start, even though it already contains superfluous crap like "dingbats" and boxes.

kingcharles 5y ago
Have emoji not become part of our writing structure though? A decent percentage of online chats and comments, especially on social networks, includes at least one emoji that couldn't be easily or accurately represented in the regular written language.
- lmm 5y ago
  Recently implementers of unicode have censored the gun emoji in a way that changes the meaning of many existing online chats and comments. So you can't easily or accurately represent things even with unicode. Emoji have never been effective or consistent at conveying meaning; at best they convey something to someone in the same subculture and time period, and often not even that. Given that unicode implementers are ok with erasing the meaning of some of them, it should be ok to eliminate more of them.
  - kingcharles 5y ago
    > Emoji have never been effective or consistent at conveying meaning; at best they convey something to someone in the same subculture and time period Isn't that the same with all words though? Think how much English usage changes in a generation. For instance, my girlfriend will use the term "I'm dead!" in a similar context to where I would say "LOL" and where my father would have said "What the fuck is loll?"
    - lmm 5y ago
      There's a spectrum. Subculture-specific slang changes quickly, but most words have a longer lifetime; reading Chaucer today is difficult but doable. Given that we don't encode words but only letters, for English you have to go back to the disappearance of þ to get a change that's relevant to text encoding. Emoji shift faster and are less effective at conveying meaning than any "real" language.
Gigachad 5y ago
This argument was lost the moment Unicode was created. Japanese carriers had created their own standard for emoji encoding for sms. And they would not switch to Unicode unless the emoji were ported over. It’s a tricky situation. Maybe allowing an arbitrary bitmap char to represent any emoji would have been better but then we could have ended up in a situation where normal text or meaningful punctuation or perhaps even fonts would get encoded as bitmaps. For something like a face or hand gesture, a bitmap likely would have been better since it would at least look the same on all platforms.
- Findecanor 5y ago
  I don't think that argument holds water. Emoji could just as well have been encoded as markup. There were for instance long-established conventions of using strings starting with : and ; . Bulletin boards extended that to a convention using letters delimited by : for example :rolleyes: . Not to mention that those codes can be typed more efficiently than browsing in an Emoji Picker box. Because emoji became characters, text rendering and font formats had to be extended to support them. There are four different ways to encode emoji in OpenType 1.8: * Apple uses embedded PNG * Google uses embedded colour bitmaps * Microsoft uses flat glyphs in different colours layered on top of one-another * Adobe and Mozilla use embedded SVG.
  - lmm 5y ago
    > Emoji could just as well have been encoded as markup. They could have, but they were already being encoded as character codepoints in existing charactersets. So any character encoding scheme that wanted to replace all use cases for existing charactersets needed to match that. If switching charactersets meant you lost the ability to use emoji until you upgraded all your applications to support some markup format, people would just not switch.
    - account42 5y ago
      > If switching charactersets meant you lost the ability to use emoji until you upgraded all your applications to support some markup format, people would just not switch. You need to upgrade those applications to support Unicode too.
saltminer 5y ago
> WTF business do emojis have in Unicode? Unicode didn't invent emoji, they incorporated it because they were already popular in Japan, and if they didn't incorporate it, it would greatly reduce Japanese adoption. Keep in mind that Unicode was intended to unify all the disparate encodings that had been brewed up to support different languages and which made exchanging documents between non-English speaking countries a nightmare. The term "mojibake" comes to mind [0] - Japan alone had so many encodings that a slang term for text encoded with something different than what your device expected (and subsequently got rendered as nonsensical/garbled text) came about. And they weren't alone, of course [1]. > What we need now is a standardized, sane subset of Unicode that implementations can support while rejecting the insane scope creep that got added on top of that. Unicode wasn't intended to be pretty. It was intended to be the one system that everyone used, and a way to increase adoption was to do some less than ideal things, like duplicate characters (so it would be easier to convert to Unicode). You may never need anything outside the BMP, but that doesn't make the rest of the planes worthless. Ignoring the value of including dead and nearing-extinct languages for preservation purposes (not being able to type a language will basically guarantee its extinction, with inventing a new encoding and storing text as jpgs being the only real alternatives), there are a lot of people speaking languages found in the SMP [2][3] ([2] has 83 million native speakers, for example). [0]: https://en.wikipedia.org/wiki/Mojibake https://en.wikipedia.org/wiki/Mojibake [1]: https://segfault.kiev.ua/cyrillic-encodings/ https://segfault.kiev.ua/cyrillic-encodings/ [2]: https://en.wikipedia.org/wiki/Modi_(Unicode_block) https://en.wikipedia.org/wiki/Modi_(Unicode_block) [3]: https://en.wikipedia.org/wiki/Chakma_(Unicode_block) https://en.wikipedia.org/wiki/Chakma_(Unicode_block)
- lmm 5y ago
  > The term "mojibake" comes to mind [0] - Japan alone had so many encodings that a slang term for text encoded with something different than what your device expected (and subsequently got rendered as nonsensical/garbled text) came about. Mojibake was not a "Japan has too many encodings" problem. It was a "western developers assume everyone is using CP1252" problem. > Unicode wasn't intended to be pretty. It was intended to be the one system that everyone used, and a way to increase adoption was to do some less than ideal things, like duplicate characters (so it would be easier to convert to Unicode). Unfortunately they undermined all that with Han Unification, with the result that it's never going to be adopted in Japan.
stubish 5y ago
You are underestimating how much language evolves. In fact, you are proposing brakes to stop if evolving. If nothing else, new currency symbols need to be incorporated every few years. The initial emoji were part of the actual writing systems of the world, even if it was relatively new and only being used by foreigners. Or maybe they have been part of world culture since the 1950s :-) ? https://en.wikipedia.org/wiki/Smiley https://en.wikipedia.org/wiki/Smiley
fomine3 5y ago
BMP was failed concept even without emoji 16bit isn't enough to contain all CJK characters.

kingcharles 5y ago

Have emoji not become part of our writing structure though? A decent percentage of online chats and comments, especially on social networks, includes at least one emoji that couldn't be easily or accurately represented in the regular written language.

lmm 5y ago
Recently implementers of unicode have censored the gun emoji in a way that changes the meaning of many existing online chats and comments. So you can't easily or accurately represent things even with unicode. Emoji have never been effective or consistent at conveying meaning; at best they convey something to someone in the same subculture and time period, and often not even that. Given that unicode implementers are ok with erasing the meaning of some of them, it should be ok to eliminate more of them.
- kingcharles 5y ago
  > Emoji have never been effective or consistent at conveying meaning; at best they convey something to someone in the same subculture and time period Isn't that the same with all words though? Think how much English usage changes in a generation. For instance, my girlfriend will use the term "I'm dead!" in a similar context to where I would say "LOL" and where my father would have said "What the fuck is loll?"
  - lmm 5y ago
    There's a spectrum. Subculture-specific slang changes quickly, but most words have a longer lifetime; reading Chaucer today is difficult but doable. Given that we don't encode words but only letters, for English you have to go back to the disappearance of þ to get a change that's relevant to text encoding. Emoji shift faster and are less effective at conveying meaning than any "real" language.

Gigachad 5y ago

This argument was lost the moment Unicode was created. Japanese carriers had created their own standard for emoji encoding for sms. And they would not switch to Unicode unless the emoji were ported over. It’s a tricky situation. Maybe allowing an arbitrary bitmap char to represent any emoji would have been better but then we could have ended up in a situation where normal text or meaningful punctuation or perhaps even fonts would get encoded as bitmaps. For something like a face or hand gesture, a bitmap likely would have been better since it would at least look the same on all platforms.

Findecanor 5y ago
I don't think that argument holds water. Emoji could just as well have been encoded as markup. There were for instance long-established conventions of using strings starting with : and ; . Bulletin boards extended that to a convention using letters delimited by : for example :rolleyes: . Not to mention that those codes can be typed more efficiently than browsing in an Emoji Picker box. Because emoji became characters, text rendering and font formats had to be extended to support them. There are four different ways to encode emoji in OpenType 1.8: * Apple uses embedded PNG * Google uses embedded colour bitmaps * Microsoft uses flat glyphs in different colours layered on top of one-another * Adobe and Mozilla use embedded SVG.
- lmm 5y ago
  > Emoji could just as well have been encoded as markup. They could have, but they were already being encoded as character codepoints in existing charactersets. So any character encoding scheme that wanted to replace all use cases for existing charactersets needed to match that. If switching charactersets meant you lost the ability to use emoji until you upgraded all your applications to support some markup format, people would just not switch.
  - account42 5y ago
    > If switching charactersets meant you lost the ability to use emoji until you upgraded all your applications to support some markup format, people would just not switch. You need to upgrade those applications to support Unicode too.

saltminer 5y ago

> WTF business do emojis have in Unicode? Unicode didn't invent emoji, they incorporated it because they were already popular in Japan, and if they didn't incorporate it, it would greatly reduce Japanese adoption. Keep in mind that Unicode was intended to unify all the disparate encodings that had been brewed up to support different languages and which made exchanging documents between non-English speaking countries a nightmare. The term "mojibake" comes to mind [0] - Japan alone had so many encodings that a slang term for text encoded with something different than what your device expected (and subsequently got rendered as nonsensical/garbled text) came about. And they weren't alone, of course [1]. > What we need now is a standardized, sane subset of Unicode that implementations can support while rejecting the insane scope creep that got added on top of that. Unicode wasn't intended to be pretty. It was intended to be the one system that everyone used, and a way to increase adoption was to do some less than ideal things, like duplicate characters (so it would be easier to convert to Unicode). You may never need anything outside the BMP, but that doesn't make the rest of the planes worthless. Ignoring the value of including dead and nearing-extinct languages for preservation purposes (not being able to type a language will basically guarantee its extinction, with inventing a new encoding and storing text as jpgs being the only real alternatives), there are a lot of people speaking languages found in the SMP [2][3] ([2] has 83 million native speakers, for example). [0]: https://en.wikipedia.org/wiki/Mojibake https://en.wikipedia.org/wiki/Mojibake [1]: https://segfault.kiev.ua/cyrillic-encodings/ https://segfault.kiev.ua/cyrillic-encodings/ [2]: https://en.wikipedia.org/wiki/Modi_(Unicode_block) https://en.wikipedia.org/wiki/Modi_(Unicode_block) [3]: https://en.wikipedia.org/wiki/Chakma_(Unicode_block) https://en.wikipedia.org/wiki/Chakma_(Unicode_block)

lmm 5y ago
> The term "mojibake" comes to mind [0] - Japan alone had so many encodings that a slang term for text encoded with something different than what your device expected (and subsequently got rendered as nonsensical/garbled text) came about. Mojibake was not a "Japan has too many encodings" problem. It was a "western developers assume everyone is using CP1252" problem. > Unicode wasn't intended to be pretty. It was intended to be the one system that everyone used, and a way to increase adoption was to do some less than ideal things, like duplicate characters (so it would be easier to convert to Unicode). Unfortunately they undermined all that with Han Unification, with the result that it's never going to be adopted in Japan.

stubish 5y ago

You are underestimating how much language evolves. In fact, you are proposing brakes to stop if evolving. If nothing else, new currency symbols need to be incorporated every few years. The initial emoji were part of the actual writing systems of the world, even if it was relatively new and only being used by foreigners. Or maybe they have been part of world culture since the 1950s :-) ? https://en.wikipedia.org/wiki/Smiley https://en.wikipedia.org/wiki/Smiley

fomine3 5y ago

BMP was failed concept even without emoji 16bit isn't enough to contain all CJK characters.