14 ms·
Emojis paved the way for UTF-8 everywhere
- whateveracct 6y agoAnd yet HN won't let me use them
- deleted 6y ago[deleted]
- grawprog 6y agoI'm glad about that. I find it weird reading through comment threads or forums and seeing mobile phone emojis scattered through. I find them distracting. I'm not really too sure why. I don't mind them in personal messages or texts and stuff, but seeing them on public pages just kind of annoys me for some reason.
- hprotagonist 6y agohow about very carefully rebased commit histories?
- Freak_NL 6y agoOddly enough nobody in my company uses emoji in commit messages, even though we have no policy that prevents it. It just doesn't make sense there. I see it on public repositories sometimes, but it never really seems to add anything useful.
- jefftk 6y agoThe amp project uses them a lot, and has a system where different kinds of commits get different leading emoji: https://github.com/ampproject/amphtml/commits/master https://github.com/ampproject/amphtml/commits/master I'm used to them at this point, and it's kind of nice when scanning commits to be able to see what type they are.
- grawprog 6y agoI have to admit, i've never actually read through any commit histories with emojis in them... Don't get me wrong, i'm not going to get mad or lose my mind or anything when I see an emoji somewhere, it just I dunno it looks wrong or something.
- mainstreem 6y agoI've seen interesting commit strategies prepending a different emoji for, e.g., feature/bug changes.
- masklinn 6y agoSwitch your system / default font to B&W if you don't like the colorised emoji.
- masklinn 6y ago> I'm glad about that. I'm not because it's completely arbitrary about it e.g. you can include 🁓, 🀕, ⏱, 🃅, box drawing, or Z̸̠̽a̷͍̟̱͔͛͘̚ĺ̸͎̌̄̌g̷͓͈̗̓͌̏̉o̴̢̺̹̕ but not trigrams, die faces, box elements, musical notes or flags. They just whitelisted/blacklisted entire blocks and called it a day. Which obviously is par for the course when it comes to HN's comment box, the markup system is even more half-assed.
- tzs 6y agoI wish they would add U+2009 (thin space) to that list. That's the standard way under the SI system to separate digit groups, e.g., 1 234 567. HN just treats it as a regular space. (The SI standard for separating the integer part from the fractional part is to use "." or ",", whichever is customary in your location. Using thin space for grouping removes the ambiguity that you get in places that use one of "."/"," for grouping and the other for a decimal point).
- jrochkind1 6y agoI wonder if that's putting things through unicode "canonical normalization", or than custom rules. Let's see what it does with `U+00BC Vulgar Fraction One Quarter Unicode Character`... ¼ Nope it allows it instead of turning into `1/4`, so that's not canonical normalization. I guess it's custom rules? Or some other unicode transformation we're not thinking of, or other third-party re-usable transformation.
- masklinn 6y ago> I guess it's custom rules? Or some other unicode transformation we're not thinking of, or other third-party re-usable transformation. They just blacklisted (or whitelisted) blocks or categories.
- jrochkind1 6y agoConverting a U+2009 THIN SPACE into an ordinary ascii space is not black/whitelisting.
- whateveracct 6y agoImagine how much more expressive my comment would've been if HN didn't strip the emoji [1] I had at the end though [1] https://emojipedia.org/pensive-face/ https://emojipedia.org/pensive-face/
- oauea 6y agoIt would make me instantly disregard your comment as immature
- whateveracct 6y agohm that feels more like a problem you have than one inherent to my comment tho
- hprotagonist 6y agoI am 100% convinced, tangentially, that mobile OS point releases use emoji as user bait to have a stronger guarantee of regular security updates. "update now to get access to :burrito: and :taco:! (also fix the following 12 CVEs that 90% of our userbase doesn't know about or read)"
- capableweb 6y ago:burrito: and :taco: are not emojis as such though, they are more alike good old "smileys" that turns characters into images. Supporting emojis would be to support the UTS#51 https://www.unicode.org/reports/tr51/ https://www.unicode.org/reports/tr51/ natively.
- hprotagonist 6y agothat's what i meant, but i was using the shortcode in my message here. "add a glyph to the emoji keyboard" is more precise.
- elFarto 6y agoI'm not sure HN will even allow emojis: Burrito: Taco: edit No, it stripped those :'(.
- madeofpalk 6y agowait they remove emojis from comments?
- seba_dos1 6y agoJust as they remove plenty of other Unicode characters here.
- deleted 6y ago[deleted]
- CodesInChaos 6y agoEmoji probably contributed to widespread support of supplemental planes (fixing systems which treated UTF-16 as UCS-2), but I doubt they contributed much to UTF-8's popularity.
- masklinn 6y ago> fixing systems which treated UTF-16 as UCS-2 Or treated UTF8 as the nonsense that MySQL's utf8 is (it's 3-bytes utf8 aka only the BMP, and silently drops anything from the first non-BMP codepoint).
- mfontani 6y agoIndeed. To ensure MySQL stores "real" UTF-8 one has to use "utf8mb4" instead of "utf8", which just rolls off the tongue and backwards (in)compatibility seems to be the reason why one can't just DWIM things backwards... "utf8mb4 or bust" it is, then!
- Freak_NL 6y agoEven more specifically, emoji paved the way for proper support of Unicode characters from beyond the Basic Multilingual Plane (BMP). There are 16 of these planes. This first block of 65,536 characters is what you can encode with only two bytes (e.g. UTF-16), and it includes most of what anyone alive needs to encode their languages adequately enough. For a long while anything encoded beyond this block had only limited support, and plenty of bugs and limitations meant that using it was tricky (well, it worked fine in LaTeX of course; via xelatex for example). This was back in 2008/2009. Characters encoded beyond the BMP in plane 1 and 2 included things like esoteric CJKV additions (East Asian ideographs) not usually in daily use, but part of historic documents. Then came the emoji additions (a core set is part of the BMP and came from Japanese telecom standards), and support is now ubiquitous. Using UTF-8 is a no-brainer for most applications, and a good things that is too!
- bawolff 6y agoYou're mixing up ucs-2 and utf-16.
- Robin_Message 6y agoTo expand on this comment, UCS-2 defines a fixed-length, 2-byte encoding of Unicode. It can therefore only represent the first 65536 characters in the Basic Multilingual Plane (BMP). UTF-16 allows representing characters outside of the BMP by using a reserved area to split a single codepoint into two surrogates that form a pair. This makes UTF-16 complicated and in some ways worse than UTF-8: the encoding is longer for many typical texts, but is still not fixed-width. The bug you typically see is that codepoints outside of the BMP are munged when clipping the text to a certain length (or reversing it, but that doesn't happen in real systems generally.)
- seba_dos1 6y agoThe reason why some older mobile phones struggle with SMS containing emojis instead of just displaying tofus in place of unsupported characters is that there's no way to send emojis in accordance to SMS standard - it defines the encoding to be UCS-2. In order to put emojis in SMS, newer phones send the messages as UTF-16 instead, technically violating the standard, which can break some parsers that only expect UCS-2 to be there.
- smrtinsert 6y agoHomesite, I miss that program. Those were the days.
- kaetemi 6y agoNot only helped to improve support for UTF-8, but also for those pesky characters that take multiple codepoints...
- masklinn 6y agoSkin tone modifiers & other composites arrived later to force supporting those properly.
- disown 6y agoPretty sure the dominance of ASCII on the internet and the efficieny/compatability of UTF-8 in relation to ASCII paved the way for UTF-8 everywhere. It is the standard unicode encoding of the internet. If anything, I would say the UTF-8 paved the way for emojis, not the other way around as the ubiquity of a unicode encoding allowed for the existence of emojis. Can't encode emojies with ASCII. You have to have unicode and its encoding first before you can have emojis.
- an_opabinia 6y agoThe ascendency of the CJK market, followed by Google Chrome, paved the way for UTF-8 everywhere. The more interesting thing is why basically no one uses Eastern ideograms in the West, except maybe the Korean ideogram for crying (ㅠㅠ) and rarely, other kaomoji-like stuff. Some kanji also tell visual stories, and most children learn them just fine, so it’s not as simple as accessibility. Borrowing kanji was also anticipated by many sci fi writers and yet is not to be.
- nneonneo 6y agoOut of curiosity, which Korean ideograph would that be? Korean doesn’t use ideograms (much) anymore; they use an alphabet packed into syllabic blocks. The character I can think of that kind of matches the description is 囧, which is a Chinese character.
- masklinn 6y ago> Out of curiosity, which Korean ideograph would that be? Yu. Having two of them looks like a crying face. Although tha (ಥ) is also a common component of crying face (ಥ_ಥ). They're talking about kaomoji which use various non-latin or fullwidth symbols (though you're right that they're largely not ideograms) to compose pretty extensive "smileys" e.g. the look of disapproval uses kannada, denko uses greek and katakana, ...
- mattnewton 6y agoShameless plug ア( ͡° ͜ʖ ͡°)ア The Gboard keyboard on android has a tab for many of these common “emoticon” faces / character sequences. If you open the emoji picker on the keyboard and then tap the far right bottom tab icon “:-)” They can get very elaborate though, these are just very basic common faces.
- masklinn 6y ago> The Gboard keyboard on android has a tab for many of these common “emoticon” faces / character sequences. If you open the emoji picker on the keyboard and then tap the far right bottom tab icon “:-)” iOS also has that on the standard Japanese "Kana" keyboard (and possibly others), under the "^_^" key.
- 1996 6y agoYes. We shouldn't dismiss the weight of non-technical people voting with their dollars. They may not care about i18n, but they do care about cute emoticons. So we get not just unicode support everywhere, but also character pickers inside the keyboard. And now we also get to benefit from unicode for things we find pretty - for example, the famous powerline https://github.com/powerline/powerline https://github.com/powerline/powerline Information density matters, and I can't wait for someone to replace "old" color coding of files (.dircolors in batch) by 1 emoticon : a music note for music files, etc.
- kens 6y agoAn entertaining article, but it's not historically accurate. If you look at measured usage, UTF-8 took off around 2005 and was the dominant web encoding by 2008. Emojis weren't added to Unicode until 2010, at which point UTF-8 usage continued to increase at exactly the same rate as before. https://en.wikipedia.org/wiki/UTF-8#/media/File:Utf8webgrowth.svg https://en.wikipedia.org/wiki/UTF-8#/media/File:Utf8webgrowt...
- crazygringo 6y agoYes, very misleading clickbait title. I was expecting some actual insightful anecdote about a pivotal choice by a major tech company that made all the difference. But nothing at all.
- redisman 6y agoDoes anyone know the real reason? Languages upgraded string to default to utf8? Browsers changed their defaults?
- larzang 6y agoDealing with the hell of mixed ISO-8859-1/15 vs CP-1252 vs UTF-8 vs simple ASCII was enough to make me an early embracer of UTF-everywhere despite mostly only having to deal with English language sources. "Someone copy/pasted from Word and broke the database again" are words you don't want to ever have to hear. UTF-8 was the sane refactoring after the initial incompatible parallel standards were established.
- MrRadar 6y agoIt's probably down to browser/OS support and the increasing importance of user-generated content on the web. Windows only got Unicode support by default on the consumer side with Windows XP which launched in 2001. Mac OS X was also released in 2001 and had Unicode support from the start (unlike Classic Mac OS which had it bolted on with poor app support). Similarly, according to this old blog post I dug up[1] Internet Explorer only got good Unicode support with 5.5 in 2000, and Netscape never had good Unicode support. Firefox, which had Unicode support from the start, was released around 2002. By 2006, which is clearly an inflection point on that graph, browsers and OSes with poor Unicode support were highly obsolete and global social media platforms like Myspace, Facebook, and Twitter, which needed Unicode to enable sharing of user posts between users writing in different languages, were on the ascendance. [1] https://www.worldtimzone.com/2002/09/24/ https://www.worldtimzone.com/2002/09/24/
- nradov 6y agoIt's interesting to watch the evolution of written language in action. I expect in 20 years we will routinely see emojis in written English novels and news articles. In 50 years we'll see them in textbooks and scientific journal articles.
- AnIdiotOnTheNet 6y agoYet another reason I'm glad for the inevitability of death. I can't be the only one who thinks emoji are a terrible idea. Granted, I also don't think logographic characters are a good idea but at least they have thousands of years of use and agreed upon semantic meaning behind them.
- szhu 6y agoIf everything you care to talk about can be easily described using thousand-year-old ideas, then I can see why you are against emojis. But this isn't true for many things people want to talk about today. Language is just an encoding for ideas, and emojis are a new compression algorithm. Using a single character, you can now convey certain thoughts and sentiments that you previously needed many more characters to reference or explain. "So then just explain it!" some might respond. "Why can't people be bothered to spend even a little time to write down what they think?" It's an accessibility issue. People have limited time every day to get their ideas across, and they deserve ways of conveying their ideas concisely. There is precedent for this too -- this is why we have acronyms and new words. "lol" and "minivan" don't have thousands of years of agreed-upon semantic meaning behind them. A final thought -- whether you think emojis are a terrible idea might not be relevant to whether they should exist. Letting people live their own lives to the fullest is much more important than making sure you, I, a future historian, or any other third party understands what they are saying. But you don't have to worry about not being to understand conversations. Given what you prefer, if someone wants to address you as a target audience, then they probably won't use emojis.
- reaperducer 6y agothis isn't true for many things people want to talk about today This makes me curious. What things can people talk about with emojis that they can't talk about in a proper language?
- DangerousPie 6y agoCan confirm, at least anecdotally. As someone who runs a website but doesn't have nearly enough time to do everything he wants to do, upgrading to UTF-8 (and specifically utf8mb4) was never a priority - until my users starting using emojis and breaking things left, right and center.
- ChrisArchitect 6y agodidn't seem like it was emojis paving the path but web-based email and internationalization of websites. Just the whole move to web in various key areas like email meant it became just less of a hair-pulling nightmare for developers to have to deal with encoding between countries and platforms. Throw in the dawn of smartphones (and emojis came along with that yes) and that was more problems on top of that, people moving between desktop/mobile/web etc. UTF-8 took care of alot of the headache.
- jcims 6y agoNow we just need emoji's for ip addresses so we can move to ipv6
- FullyFunctional 6y ago"And it's amusing to see Apple using new emojis as a carrot to get people to install the latest security patches." OMG, that makes so much sense. I was the opposite, grumbling about not caring about that silliness, not realizing the psychology of things. Having suffered through the dark ages with Microsoft increasingly ruining the world with an endless stream of proprietary crap (I still hate them for making people think tab width is configurable), it's amazing to step back and witness how much things have improved (on this narrow slice).
- Andrew_nenakhov 6y agoIn russian segment of internet, various cyrillic encodings (win, dos, mac, koi-8) were a huge problem, and only UTF-8 finally solved it long before emojis became a thing. It is known that developers from English-speaking countries are generally oblivious to encoding problems. Probably they could get by on ASCII far longer than the rest of the world, so no wonder that they might confuse cause and effect in this case.
- qwerty456127 6y agoDo e-mail clients like Thunderbird and Outlook already use UTF-8 by default? The last time I checked they used ANSI/ISO codepages for new E-mails. I mean the desktop MS Office Outlook app primarily, but I'm curious about the web apps as well.
- amelius 6y agoWhat I don't like about this approach is that everybody potentially sees different versions of the emoji. (And by the way, HN stripped out the emoji I put in this comment; perhaps that is for the better, but it's kind of funny that the software we use here is quite opposite to the software we make in our day jobs)
- kevsim 6y agoEmojis are what caused me to learn about unicode personally. We had a big bloated library handling emoji stuff for us in our product [0]. Then one day, someone said that they wanted the comment threads to behave more like Slack - that if only an emoji was typed in a comment (without any additional text), it should appear big, otherwise it should be small. This required detecting when text was only an emoji which required me to learn how the heck this stuff actually worked. Great experience and I'm forever indebted to emojis. 0: https://kitemaker.co https://kitemaker.co, an awesome new issue tracker with tons of hotkeys (and emojis)