9 ms·
Emoji Under the Hood
- breck 6y agoI thought I knew Emoji, but there was a lot I didn’t know. Thank you, a very enjoyable and enlightening read. Also, “dingbats”! I rarely seen that word since I was a kid (when I had no idea what that voodoo was but loved it).
- yuntei 6y agoand now to see how emoji rendering is completely broken, put a gear u+2699 text variant and emoji variant in some html and set the font to menlo in one element, and monaco in another element and then view it in chrome, safari desktop, and safari ios, and also select and right click on it in chrome, and maybe also post it into the comment section of various websites. Every single combination of text variant and emoji variant will be displayed in complete randomness :)
- ijidak 6y agoThis is eye opening. So many frustrations I've had with emoji over the years is explained via this post. Big thank you to the OP.
- chronogram 6y agoWhat kind of frustrations?
- devadvance 6y agoFantastic post that builds up knowledge along the way. A fun case where this type of knowledge was relevant: when creating emoji short links with a couple characters (symbols), I made sure to snag both URLs: one with the emoji (codepoint + `U+FE0F`) and one with just the symbol codepoint. Another thing worth calling out: you can get involved in emoji creation and Unicode in general. You can do this directly, or by working with groups like Emojination [0]. [0] http://www.emojination.org/ http://www.emojination.org/
- codetrotter 6y agoThe emojination website mentions UTC and ESC. UTC in this context certainly means Unicode Technical Committee. And after a bit of Googling it seems that ESC is the Unicode Emoji Subcommittee. Some of the suggested emojis are marked as UTC rejected, some as ESC rejected or ESC pushback. Does it mean that both UTC and ESC has to approve each suggested emoji? And is there a place to see the reason for rejection and a place to see what kind of pushback they are receiving?
- dgellow 6y agoIt's for "Emoji SubCommittee" (aka ESC). > Unicode Emoji Subcommittee: > The Unicode Emoji Subcommittee is responsible for the following: > - Updating, revising, and extending emoji documents such as UTS #51: Unicode Emoji and Unicode Emoji Charts. > - Taking input from various sources and reviewing requests for new emoji characters. > - Creating proposals for the Unicode Technical Committee regarding additional emoji characters and new emoji-related mechanisms. > - Investigating longer-term mechanisms for supporting emoji as images (stickers). From https://unicode.org/emoji/techindex.html https://unicode.org/emoji/techindex.html Edit: Welp, the parent comment was asking what "ESC" stands for, but has now been updated, so this comment is now outdated :)
- codetrotter 6y agoSorry, yeah I was originally asking about what ESC stands for but found some info shortly after and updated my comment. But I appreciate the additional info anyways :)
- lifthrasiir 6y agoIt's complicated. So this mainly boils down to the relationship between UTC and ESC. ESC contributes to UTC, along with other groups (e.g. Scripts Ad Hoc Group or IRG) or other individuals (you can submit documents to UTC [1]), and technically UTC has a right to reject ESC contributions. In reality however ESC manages a huge volume of emoji proposals to UTC and distills them down to a packaged submission, so UTC rarely outright rejects ESC contributions. After all ESC is a part of UTC so there is a huge overlap anyway (e.g. Mark Davis is the Unicode Consortium and ESC chair). "UTC rejected" emojis thus generally come from the direct proposal to UTC. You can see a list of emoji requests [2] but it lacks much information. This lack of transparency in the ESC process is well known and was most directly criticized by contributing experts in 2017 [3]. ESC responded [4] that there are so many flawed proposals (with no regards to the submission criteria [5]) that it is infeasible to document all of them. IMHO it's not a very satisfactory answer, but still understandable. [1] https://www.unicode.org/L2/ https://www.unicode.org/L2/ [2] https://www.unicode.org/emoji/emoji-requests.html https://www.unicode.org/emoji/emoji-requests.html [3] https://www.unicode.org/L2/L2017/17147-emoji-subcommittee.pdf https://www.unicode.org/L2/L2017/17147-emoji-subcommittee.pd... [4] https://www.unicode.org/L2/L2017/17192-response-cmts.pdf https://www.unicode.org/L2/L2017/17192-response-cmts.pdf [5] https://www.unicode.org/emoji/proposals.html https://www.unicode.org/emoji/proposals.html
- mannerheim 6y ago> Currently they are used for these three flags only: England, Scotland and Wales: Not quite true, you can get US state flags with this as well.
- petepete 6y agoI've never seen them use, have they actually been implemented by any of the creators?
- mannerheim 6y agoIf I type the following into ghci, I get the state flag of Texas: putStrLn "\x1f3f4\xe0075\xe0073\xe0074\xe0078\xe007f" The first character is a flag, the last character is a terminator, and in between are the tag characters corresponding to the ASCII for ustx. Just take those characters and subtract 0xe0000 from them, 0x75, 0x73, 0x74, 0x78. https://en.wikipedia.org/wiki/Tags_(Unicode_block) https://en.wikipedia.org/wiki/Tags_(Unicode_block) Edit: Just for fun: import Data.StateCodes import Data.Char putStrLn $ map (map toLower . show . snd) allStates >>= \stateCode -> '\x1f3f4':map (toEnum . (0xe0000+) . fromEnum) ("us" ++ stateCode) ++ "\xe007f"
- azornathogron 6y agoOh my god they've put terminal escape codes into Unicode.
- petepete 6y agoYeah I understand how it works but none of the major vendors have actually implemented the emoji. https://emojipedia.org/flag-for-texas-ustx/ https://emojipedia.org/flag-for-texas-ustx/
- mannerheim 6y agoTouché.
- Sniffnoy 6y ago
- mojuba 6y agoCan someone explain, what are the rules for substring(m, n) given all the madness that's today's Unicode? Is it standardized or it's up to the implementations?
- _ZeD_ 6y agoit think the only resonable rule for substring(m, n) is "don't"
- mojuba 6y agoSo string is no longer a "string of characters", it is in fact a program (not Turing complete) that you need to execute. Though substring(m, n) still makes sense in at least interactive text manipulation: how do you do copy/paste?
- techdragon 6y agoNot really. A Unicode string is more like a sequence of data built from simple binary structs, which belong to a smallish group of valid structs. Additionally, some but not all, of these structs can be used to infer the validity of subsequent structs in the sequence if your parsing in a more byte-at-a-time fashion. Alternately if your happy dealing with a little less forward compatibility and go for explicit enumeration of all groups of valid bytes you can be a lot more sure of things but it’s harder to make this method as performant as the byte-at-a-time method, which given the complete ubiquity of string processing in software... leads to the dominance of the byte-at-a-time method.
- truefossil 6y agoThe safest path is to consider it a blob. There is some library that can render it magically and that's the only wise thing you can do. The internal structure is hard to understand. Also, definitions change over time. So, you better leave it all to professionals.
- spookthesunset 6y ago
- remux 6y agoGreat post!
- lifthrasiir 6y ago> One weird inconsistency I’ve noticed is that hair color is done via ZWJ, while skin tone is just modifier emoji with no joiner. Why? Seriously, I am asking you: why? I have no clue. Mainly because skin tone modifiers [1] predate the ZWJ mechanism [2]. For hair colors there were two contending proposals [3] [4], one of which doesn't use ZWJ, and the ZWJ proposal was accepted because new modifiers (as opposed to ZWJ sequences) needed the architectural change [5]. [1] https://www.unicode.org/L2/L2014/14213-skin-tone-mod.pdf https://www.unicode.org/L2/L2014/14213-skin-tone-mod.pdf [2] https://www.unicode.org/L2/L2015/15029r-zwj-emoji.pdf https://www.unicode.org/L2/L2015/15029r-zwj-emoji.pdf [3] https://www.unicode.org/L2/L2017/17082-natural-hair-color.pdf https://www.unicode.org/L2/L2017/17082-natural-hair-color.pd... [4] https://www.unicode.org/L2/L2017/17193-hair-colour-proposal.pdf https://www.unicode.org/L2/L2017/17193-hair-colour-proposal.... [5] https://www.unicode.org/L2/L2017/17283-response-hair.pdf https://www.unicode.org/L2/L2017/17283-response-hair.pdf
- kevincox 6y agoRandal Monroe was also wondering why most of the emoji aren't just modifiers: https://xkcd.com/1813/ https://xkcd.com/1813/
- vanderZwan 6y agoI wonder how many years it'll take for someone to train a neural network to generate emojis for all possible modifiers, regardless of whether they're currently real combinations.
- blauditore 6y agoIntuitively, I think this would be doable today with style transfer networks. But maybe there wouldn't be enough training data with existing emojis. I hope someone who knows more about it can tune in...
- theseanz 6y agoThere's Emoji Mashup Bot+ - "Tries to create new emojis out of three random emoji parts" https://twitter.com/emojimashupplus https://twitter.com/emojimashupplus
- itsmeamario 6y agoGreat quality post. I'd like to see more things like this on HN. Interesting and I learnt a lot about emojis and UTF.
- truefossil 6y agoI wonder why Mediterranean nations switched from ideograms to alphabet as soon as one was invented. Probably they did not have enough surplus grain to feed something like the Unicode consortium?
- meepmorp 6y agoHieroglyphics weren't really ideographic after a very early point, because it's a pain in the ass making up new symbols for every word. Very quickly, it transitioned to being largely an abjad, representing only consonants. Abjads work reasonably well for semitic languages, as the consonantal roots of words carry the meaning and a reader would be able to fill in the vowels themselves via context. According to the account I've heard, it's the greeks who invented the alphabet, by accident. The Phoenician script used single symbols to represent consonants, including the glottal stop (and some pharyngeal consonant that would likely be subject to a similar process, iirc). The glottal stop was represented by aleph, and because Greek didn't have contrastive glottal stops in its phoneme inventory, Greeks just interpreted the vowel that followed it as what the symbol was meant to represent. It's a bit of a just so story, but also completely plausible.
- kps 6y agoAn alphabet (or syllabary, abjad, abugida) has a small set of symbols that can express anything, which means that it could be used by people who did something other than read and write for a living. Probably no accident that the first to catch on, and the root of possibly all others, was spread by Phoenician traders.
- tomduncalf 6y agoReally interesting and well written (and entertaining!) post. I was vaguely aware of most of it but hadn’t appreciated how the ZWJ system for more complex emojis made up of basin ones means the meaning can be discerned even if your device doesn’t support the new emoji, clever approach!
- woko 6y ago> Unicode allocates 2²¹ (~2 mil) characters called codepoints. Sorry, programmers, but it’s not a multiply of 8 . Why would 2^21 not be a multiple of 2^3?
- RedNifre 6y agoIt's a typo, they meant ~2²¹ instead of 2²¹, because it's 17*2^16, which is more like ~2^20.087. (And that's not even true either, since a couple values like FFFF are forbidden)
- deleted 6y ago[deleted]
- howtodowtle 6y agoOf course, 17 x 2^16 is also a multiple of 2^3: 17 x 2^16 = 17 x 2^13 x 2^3 (reposted/edited because * was interpreted as formatting)
- RedNifre 6y agoIn case hacker news doesn't show emoji, I meant m( Right, I guess I was thinking more of "not a power of 2" instead of "not a multiple of 8". On second thought, the author might have meant "Sorry that the exponent is not a multiple of 8" as in Unicode neither has 2^16 nor 2^32 code points.
- kps 6y agoAgreed; they meant “not a power of 2⁸”.
- Hawzen 6y ago> The most popular encoding we use is called Unicode Unicode is a character set, not an encoding UTF-8, UTF-16, etc. are encodings of that character set
- imtiyaz 6y agoNever gone to these nitty gritties. Very well explained. Thanks Nikita.
- tonsky 6y agoYou are welcome! Glad you liked it
- peteretep 6y agoAn excellent article, although: > “Ü” is a single grapheme cluster, even though it’s composed of two codepoints: U+0055 UPPER-CASE U followed by U+0308 COMBINING DIAERESIS. would be a great opportunity to talk about normal form, because there’s also a single code point version: “latin capital letter u with diaeresis”.
- colejohnson66 6y agoDoes anyone know the history behind why there’s two ways to “encode” things like that? What’s the rationale for having both combining and precombined codepoints?
- bombcar 6y agoI believe a lot of the "combined" characters are (basically) from importing old codepages directly into Unicode, and they did that so it would be a simple formula to convert from the various codepages in use. I may be wrong however.
- MrGilbert 6y agoReading about the 2 million codepoints: Is there a good set of open-source licensed fonts which cover as many codepoints as possible? Just curiosity, no real usecase at the moment. I don't think it would make sense to create one huge font for this, right?
- pta2002 6y agoGoogle's Noto Fonts[1] attempt to cover all of Unicode and are released under the SIL Open Font License. [1] https://www.google.com/get/noto/ https://www.google.com/get/noto/
- MrGilbert 6y agoThat looks incredible complete, thank you!
- dan-robertson 6y agoThere’s a project called, I think, gnufont but their font is a bitmap font...
- MrGilbert 6y agoAh, thank you! Searching for "gnufont" brought me to[1], which looks pretty nice indeed. [1]: https://www.gnu.org/software/freefont/ https://www.gnu.org/software/freefont/
- dan-robertson 6y agoI think that’s what I was thinking of. I guess they’ve got some vector outlines now
- BlueGh0st 6y agoI wish I could read this without getting a migraine. The "darkmode" joke was funny until I realized there was no actual way to turn it on.
- tobz1000 6y agohttps://darkreader.org/ https://darkreader.org/
- sundarurfriend 6y agoI just turned this off today, after one too many "an extension is slowing this page down" warnings from Firefox, always from Dark Reader. It's a pretty useful addon, but there's enough websites that implement their own dark mode that it's less necessary these days (I hope), and possibly making it not worth the slowdown.
- jffry 6y agoFirefox's reader mode works great and includes a dark theme. The icon shows up in the right side of the URL bar, but you can always force it by prepending the URL, e.g. about:reader?url=<url>
- vlmutolo 6y agoI always edit the CSS style when this site comes up
- vanderZwan 6y ago> Flags don’t have dedicated codepoints. Instead, they are two-letter ligatures. (...) There are 258 valid two-letter combinations. Can you find them all? Well this nerd-sniped me pretty hard https://next.observablehq.com/@jobleonard/which-unicode-flags-are-reversible https://next.observablehq.com/@jobleonard/which-unicode-flag... That was a fun little exercise, but enough time wasted, back to work.
- mercer 6y agoHaha, playing around with reversing flags was the first thing I thought about trying.
- vanderZwan 6y agoThe surprising result (to me at least) was that out of 270 valid letter combinations, 105 can be reversed. The odd number is easy to explain: letter pairs like MM => MM can add a single flag instead of a pair of two flags, but the fact that almost two out of every five flags are reversible feels pretty high to me.
- SamBam 6y ago> but the fact that almost two out of every five flags are reversible feels pretty high to me. I think some letter-frequency analysis can probably explain it. Given the fact that certain letters are less likely both as the first slot and second (e.g., there are only 4 country codes that start with J, and 3 that end with J), the letters that can be used as both first and second characters are over-represented. It's the same as how far more English language words can be reversed to make other valid words than you would expect if the letters were equally-frequent and arbitrarily arranged.
- aglionby 6y agoGreat post, entertainingly written. Back in 2015, Instagram did a blog post on similar challenges they came across implementing emoji hashtags [1]. Spoiler alert: they programmatically constructed a huge regex to detect them. [1] https://instagram-engineering.com/emojineering-part-ii-implementing-hashtag-emoji-7b653b221c82 https://instagram-engineering.com/emojineering-part-ii-imple...
- lifthrasiir 6y agoNowadays you can refer to UAX #31 for hashtag identifiers (first specified in 2016): https://www.unicode.org/reports/tr31/#hashtag_identifiers https://www.unicode.org/reports/tr31/#hashtag_identifiers
- rkangel 6y agoThe article is great, but there is one slightly misleading bit at the start: > The most popular encoding we use is called Unicode, with the two most popular variations called UTF-8 and UTF-16. Unicode is a list of codepoints - the characters talked about in the rest of the article. These live in a number space that's very big (~2^23 as discussed). You can talk about these codepoints in the abstract as this article does, but at some point you need to put them in a computer - store them on disk or transmit them over a network connection. To do this you need a way to make a stream of bytes store a series of unicode codepoints. This is an 'encoding', UTF-8 and UTF-16, UTF-32 etc. are different encodings. UTF-32 is the simplest and most 'obvious' encoding to use. 32 bits is more than enough to represent every codepoint, so just use a 32-bit value to represent each codepoint, and keep them in a big array. This has a lot of value in simplicity, but it means that text ends up taking up a lot of space. Most western text (e.g. this page) fits in the first 127 bits and so for the majority of values, most of the bits will be 0. UTF-16 is an abomination that is largely Microsoft's fault and is the default unicode encoding on Windows. It is based on the fact that most text in most language fits in the first 65535 unicode codepoints - referred to as the 'Basic Multilingual Plane'. This means that you can use a 16 bit value to represent most codepoints, so unicode is stored as an array of 16-bit values ("wide strings" in MS APIs). Obviously not all Unicode values fit in, so there is the capability to use two UTF-16 values to represent a code-point. There are many problems with UTF-16, but my favourite is that it really helps you to have 'unicode surprises' in your code. Something in your stack that assumes single byte characters and barfs on higher unicode values is well known, and you find it in testing fairly often. Because UTF-16 is a single value for the vast majority of normal codepoints, it makes that worse by making it only happen in a very small number of cases that you will inevitably only discover in production. UTF-8 is the generally agreed to be the best encoding (particularly among people who don't work for Microsoft). It is a full variable length encoding, so a single codepoint can take 1, 2, 3 or 4 bytes. It has lots of nice properties, but one is that codepoints that are <= 127 encode using a single byte. This means that proper ASCII is valid UTF-8.
- rectang 6y agoFor people who want to hear more on this subject I gave a talk for Papers We Love Seattle on UTF-8, its origins and evolution, and how it compares against other encodings: https://www.youtube.com/watch?v=mhvaeHoIE24 https://www.youtube.com/watch?v=mhvaeHoIE24 "Smiling Cat Face With Heart Eyes Emoji" plays a major role. :) It doesn't cover the same ground as this wonderful post with its study of variation selectors and skin-tone modifiers, but it provides the prerequisites leading up to it. > UTF-16 is an abomination that is largely Microsoft's fault I think that's unfair. The problem lies more in the conceptualization of "Unicode" in the late 1980s as a two-byte fixed-width encoding whose 65k-sized code space would be enough for the characters of all the world's living languages. (I cover that here: https://www.youtube.com/watch?v=mhvaeHoIE24&t=7m10s https://www.youtube.com/watch?v=mhvaeHoIE24&t=7m10s ) It turns out that we needed more space, and if Asian countries had had more say from the start, it would have been obvious earlier that a problem existed.
- z3t4 6y agoRelated: implementing Emoji support in a text editor: https://xn--zta-qla.com/en/blog/editor10.htm https://xn--zta-qla.com/en/blog/editor10.htm
- mshenfield 6y agoIt's a post about emojis, but I feel like I understand Unicode better now?
- artur_makly 6y agoWhat I really want to know is the story behind how these emoji's came to be?! Who was tasked to come up with this sticker list of symbols? What was the decision/strategy behind the selection of these symbols? etc etc. it seems soooo arbitrary at first-glance. And how do we as a community propose new icons while considering others to be removed/replaced?
- rynt 6y ago99PI did a story on the process of submitting a new emoji request to the Unicode Consortium that you might find interesting: https://99percentinvisible.org/episode/person-lotus-position/ https://99percentinvisible.org/episode/person-lotus-position...
- artur_makly 6y agobrilliant. thank you {{ U+1F64F }}
- kaeruct 6y agoI'm confused about the part saying flags don't work on Windows because I can see them on Firefox (on Windows). They don't work on Edge though.
- tonsky 6y agoI guess FF ships its own version
- avipars 6y agoReally interesting article, why haven't platforms banned Ų̷̡̡̨̫͍̟̯̣͎͓̘̱̖̱̣͈͍̫͖̮̫̹̟̣͉̦̬̬͈͈͔͙͕̩̬̐̏̌̉́̾͑̒͌͊͗́̾̈̈́̆̅̉͌̋̇͆̚̚̚͠ͅ or figured out a way to parse/contain the character to it's container?
- Robizzle01 6y agoLove it, thanks for writing it up. Regarding Windows and flags, I heard it was a geopolitical issue. Basically, to support flag emoji you’d have to decide whether or not to recognize some states (e.g. Taiwan) which can anger other states. Not sure if that’s the real reason or not. A couple questions I still have: 1. Why make flags multiple code points when there’s plenty of unused address space to assign a single code point? 2. Any entertaining backstories regarding platform specific non-standard emoji, such as Windows ninja cat (https://emojipedia.org/ninja-cat/ https://emojipedia.org/ninja-cat/)? Why would they use those code points rather than ? 3. Is it possible to modify Windows to render emoji using Apple’s font (or a modified Segue that looks like Apple’s)? 4. Which emoji look the most different depending on platform? Are there any that cause miscommunication? 5. Do any glyphs render differently based on background color, e.g. dark mode?
- bewuethr 6y agoThis is a really nice overview! I have one nit about an omission: in addition to the emoji presentation selector, FE0F, which forces "presentation as emoji", there's also the text presentation selector, FE0E, which does the opposite [1]. The Emoji_Presentation property [2] determines when either is required; code points with both an emoji and a text presentation and the property set to "Yes" default to emoji presentation without a selector and require FE0E for text presentation; code points with the property set to "No" default to text presentation and require FE0F for emoji presentation. There's a list [3] with all emoji that have two presentations, and the first three rows of the Default Style Values table [4] shows which emoji default to which style. [1]: https://unicode.org/reports/tr51/#Emoji_Variation_Sequences https://unicode.org/reports/tr51/#Emoji_Variation_Sequences [2]: http://unicode.org/reports/tr51/#Emoji_Properties_and_Data_Files http://unicode.org/reports/tr51/#Emoji_Properties_and_Data_F... [3]: https://unicode.org/emoji/charts/emoji-variants.html https://unicode.org/emoji/charts/emoji-variants.html [4]: https://unicode.org/emoji/charts-13.0/emoji-style.html https://unicode.org/emoji/charts-13.0/emoji-style.html
- zimpenfish 6y agoApropos of nothing, macOS 11.3 beta and iOS 14.5 do support the combined emojis near the bottom - instead of <heart><fire>, I actually get <flaming heart> as expected.