8 ms·

If you think the Unicode flag emoji take a lot of bytes, then consider the family emoji! (https://unicode.org/emoji/charts/full-emoji-list.html#family https://u

If you think the Unicode flag emoji take a lot of bytes, then consider the family emoji! (https://unicode.org/emoji/charts/full-emoji-list.html#family https://unicode.org/emoji/charts/full-emoji-list.html#family) I'm in the process of designing a scripting language and implementing it in C++. I plan to put together a YouTube series about it. (Doesn't everyone want to see Bison and Flex mixed with proper unit tests and C++20 code?) Due to my future intended use case, I needed good support for Unicode. I thought that I could write it myself, and I was wrong. I wasted two weeks (in my spare time, mostly evenings) trying to cobble together things that should work, identifying patterns, figuring out how to update it as Unicode itself is updated, thinking about edge cases, i18n, zalgo text, etc. And then I finally reached the point where I knew enough to know that I was making the wrong choice. I'm now using ICU. (https://icu.unicode.org/ https://icu.unicode.org/) It's huge, it was hard to get it working in my environment, and there are very few examples of it's usage online, but after the initial setup dues are paid, it WORKS. Aside: Yes, I know I'm crazy for implementing a programming language that I intend for serious usage. Yes, I have good reasons for doing it, and yes I have considered alternatives. But it's fun, so I'm doing it anyways. Moral of the story: Dealing with Unicode is hard, and if you think it shouldn't be that hard, then you probably don't know enough about the problem!

Gigachad 5y ago
It always feels like the most amount of work goes to the least used emoji. So many revisions and additions to the family emoji and yet it’s one of the ones I don’t recall anyone ever using. I think the trap Unicode got in to is technically they can have infinite emoji so they just don’t ever have a way to say no to new proposals.
- laumars 5y ago
  They do say no though. Frequently too. The problem with Unicode is simply that it’s trying to solve a very hard problem.
  - tialaramex 5y ago
    Exactly this. Humans have incredibly complicated writing systems, and all Unicode wants to do is encode them all. Keep in mind that the trivial toy system we're more familiar with, ASCII, already has some pretty strange features because even to half-arse one human writing system they needed those features. Case is totally wild, it only applies to like 5% of the symbols in ASCII, but in the process it means they each need two codepoints and you're expected to carry around tech for switching back and forth between cases. And then there are several distinct types of white space, each gets a codepoint, some of them try to mess with your text's "position" which may not make any sense in the context where you wanted to use it. What does it mean to have a "horizontal tab" between two parts of the text I wanted to draw on this mug? I found a document which says it is the same as "eight spaces" which seems wrong because surely if you wanted eight spaces you'd just write eight spaces. And after all that ASCII doesn't have working quotation marks, it doesn't understand how to spell a bunch of common English words like naïve or café, pretty disappointing.
    - xxpor 5y ago
      >Humans have incredibly complicated writing systems Not only that, there isn't even agreement about what's correct all the time! >it doesn't understand how to spell a bunch of common English words like naïve or café, pretty disappointing. A perfect example of this, since I would argue English doesn't have any diacritics at all. So the use of café is code switching. :)
      - mattkrause 5y ago
        Not a New Yorker writer, I see....
        xxpor 5y ago
        Thank God! (seriously, every time I read an article from them I get distracted/nerd sniped by their diaeresis use...)
      - brimble 5y ago
- masklinn 5y ago
  > It always feels like the most amount of work goes to the least used emoji. I always feel like those emoji were added on purpose in order to force implementations to fix their unicode support. Before emoji were added, most software had completely broken support for anything beyond the BMP (case study: MySQL's so-called "UTF8" encoding). The introduction of emoji, and their immediate popularity, forced many systems to better support astral planes (that is officially acknowledged: https://unicode.org/faq/emoji_dingbats.html#EO1 https://unicode.org/faq/emoji_dingbats.html#EO1) Progressively, emoji using more advanced features got introduced, which force systems (and developers) to fix their unicode-handling, or at least improve it somewhat e.g. skintones for combining codepoints, etc.... > I think the trap Unicode got in to is technically they can have infinite emoji so they just don’t ever have a way to say no to new proposals. You should try to follow a new character through the process, because that's absolutely not what happens and shepherding a new emoji through to standardisation is not an easy task. The unicode consortium absolutely does say no, and has many reasons to do so. There's an entire page on just proposal guidelines (https://unicode.org/emoji/proposals.html https://unicode.org/emoji/proposals.html), and following it does not in any way ensure it'll be accepted.
  - mike_hock 5y ago
    WTF business do emojis have in Unicode? The BMP is all there ever should have been. Standardize the actual writing systems of the world, so everyone can write in their language. And once that is done, the standard doesn't need to change for a hundred years. What we need now is a standardized, sane subset of Unicode that implementations can support while rejecting the insane scope creep that got added on top of that. I guess the BMP is a good start, even though it already contains superfluous crap like "dingbats" and boxes.
    - kingcharles 5y ago
      Have emoji not become part of our writing structure though? A decent percentage of online chats and comments, especially on social networks, includes at least one emoji that couldn't be easily or accurately represented in the regular written language.
- jonas21 5y ago
  This work wasn't done for emoji. They use the same zero-width joiner character [1] that exists to support Indic scripts like Devanagari, and any system that properly handles these languages will also properly handle the emoji. Yes, this adds a lot of complexity, but it's really a question of whether that complexity is justified in order to support all of the world's languages. And I think many would argue that it is. [1] https://en.wikipedia.org/wiki/Zero-width_joiner https://en.wikipedia.org/wiki/Zero-width_joiner
  - jahewson 5y ago
    Emoji has turned out to be a great way to enhance complex script support in much real-world software and they get my full support for that reason too.
josephg 5y ago
Handling unicode can be fine, depending on what you're doing. The hard parts are: - Counting, rendering and collapsing grapheme clusters (like the flag emoji) - Converting between legacy encodings (shiftjis, ko8, etc) and UTF-8 / UTF-16 - Canonicalization If all you need is to deal with utf8 byte buffers, you don't need all that stuff. And your code can stay simple, small and fast. IIRC the rust standard library doesn't bother supporting any of the hard parts in unicode. The only real unicode support in std is utf8 validation for strings. All the complex aspects of unicode are delegated to 3rd party crates. By contrast, nodejs (and web browsers) do all of this. But they implement it in the same way you're suggesting - they simply call out to libicu.
- tialaramex 5y ago
  > The only real unicode support in std is utf8 validation for strings. Rust's core library gives char methods such as is_numeric which asks whether this Unicode codepoint is in one of Unicode's numeric classes such as the letter-like-numerics and various digits. (Rust does provide char with is_ascii_digit and is_ascii_hexdigit if that's all you actually cared about) So yes, the Rust standard library is carrying around the entire Unicode standard class rule list among other things, of course Rust's library isn't built to a vast binary, so if you never use these features your binary doesn't get that code.
amelius 5y ago
Why is this stuff even reinvented for every programming language? Isn't it about time that we have some common language that every other language builds on?
- IncRnd 5y ago
  So said every writer of a standard immediately before writing another standard to replace all others.
  - rwbhn 5y ago
    Obligatory xkcd https://xkcd.com/927/ https://xkcd.com/927/
- lmm 5y ago
  ICU is out there with bindings in many languages. People who know what they're doing use ICU.
- johndough 5y ago
  > Isn't it about time that we have some common language that every other language builds on? That language is C. It is debatable whether it was a good choice, but at least this is how it turned.
DecoPerson 5y ago
Do you have a YouTube for people to subscribe to in anticipation of you releasing your YouTube series about your work? The development processes of new languages is so intriguing.
- coreyp_1 5y ago
  I'll post about it here on HN after I have a few episodes up.
dagmx 5y ago
It would actually be pretty interesting to see how you use Bison and Flex with utf-8. Most resources say to not bother due to lack of support for Unicode, but they're so ubiquitous
- account42 5y ago
  Do they need special support for UTF-8? One of the nice things about UTF-8 is that you can treat it as an 8-bit encoding in many cases if you only care about substrings and don't need to decode individual non-ASCII characters.
  - dagmx 5y ago
    At some point in the pipeline you need the tooling to constrain identifiers to XID_START and XID_CONTINUE
Vindicis 5y ago
I know how that feels. I wrote a little c++ program to fetch data in Unicode from a dB and then normalize it to ascii to be used for analytic purposes. A lot faster to do it on ascii than trying to handle all the fun cases of how many ways can an e etc... be input. ICU to the rescue! Took a couple weeks of getting up to speed as ICU itself wasn't too bad to figure out. But, you find out very quickly that to use it, you need to have a good understanding of a number of the Unicode technical reports to actually understand how to make use of it. Fun times indeed.

Gigachad 5y ago

It always feels like the most amount of work goes to the least used emoji. So many revisions and additions to the family emoji and yet it’s one of the ones I don’t recall anyone ever using. I think the trap Unicode got in to is technically they can have infinite emoji so they just don’t ever have a way to say no to new proposals.

laumars 5y ago
They do say no though. Frequently too. The problem with Unicode is simply that it’s trying to solve a very hard problem.
- tialaramex 5y ago
  Exactly this. Humans have incredibly complicated writing systems, and all Unicode wants to do is encode them all. Keep in mind that the trivial toy system we're more familiar with, ASCII, already has some pretty strange features because even to half-arse one human writing system they needed those features. Case is totally wild, it only applies to like 5% of the symbols in ASCII, but in the process it means they each need two codepoints and you're expected to carry around tech for switching back and forth between cases. And then there are several distinct types of white space, each gets a codepoint, some of them try to mess with your text's "position" which may not make any sense in the context where you wanted to use it. What does it mean to have a "horizontal tab" between two parts of the text I wanted to draw on this mug? I found a document which says it is the same as "eight spaces" which seems wrong because surely if you wanted eight spaces you'd just write eight spaces. And after all that ASCII doesn't have working quotation marks, it doesn't understand how to spell a bunch of common English words like naïve or café, pretty disappointing.
  - xxpor 5y ago
    >Humans have incredibly complicated writing systems Not only that, there isn't even agreement about what's correct all the time! >it doesn't understand how to spell a bunch of common English words like naïve or café, pretty disappointing. A perfect example of this, since I would argue English doesn't have any diacritics at all. So the use of café is code switching. :)
    - mattkrause 5y ago
      Not a New Yorker writer, I see....
      - xxpor 5y ago
        Thank God! (seriously, every time I read an article from them I get distracted/nerd sniped by their diaeresis use...)
    - brimble 5y ago
masklinn 5y ago
> It always feels like the most amount of work goes to the least used emoji. I always feel like those emoji were added on purpose in order to force implementations to fix their unicode support. Before emoji were added, most software had completely broken support for anything beyond the BMP (case study: MySQL's so-called "UTF8" encoding). The introduction of emoji, and their immediate popularity, forced many systems to better support astral planes (that is officially acknowledged: https://unicode.org/faq/emoji_dingbats.html#EO1 https://unicode.org/faq/emoji_dingbats.html#EO1) Progressively, emoji using more advanced features got introduced, which force systems (and developers) to fix their unicode-handling, or at least improve it somewhat e.g. skintones for combining codepoints, etc.... > I think the trap Unicode got in to is technically they can have infinite emoji so they just don’t ever have a way to say no to new proposals. You should try to follow a new character through the process, because that's absolutely not what happens and shepherding a new emoji through to standardisation is not an easy task. The unicode consortium absolutely does say no, and has many reasons to do so. There's an entire page on just proposal guidelines (https://unicode.org/emoji/proposals.html https://unicode.org/emoji/proposals.html), and following it does not in any way ensure it'll be accepted.
- mike_hock 5y ago
  WTF business do emojis have in Unicode? The BMP is all there ever should have been. Standardize the actual writing systems of the world, so everyone can write in their language. And once that is done, the standard doesn't need to change for a hundred years. What we need now is a standardized, sane subset of Unicode that implementations can support while rejecting the insane scope creep that got added on top of that. I guess the BMP is a good start, even though it already contains superfluous crap like "dingbats" and boxes.
  - kingcharles 5y ago
    Have emoji not become part of our writing structure though? A decent percentage of online chats and comments, especially on social networks, includes at least one emoji that couldn't be easily or accurately represented in the regular written language.
jonas21 5y ago
This work wasn't done for emoji. They use the same zero-width joiner character [1] that exists to support Indic scripts like Devanagari, and any system that properly handles these languages will also properly handle the emoji. Yes, this adds a lot of complexity, but it's really a question of whether that complexity is justified in order to support all of the world's languages. And I think many would argue that it is. [1] https://en.wikipedia.org/wiki/Zero-width_joiner https://en.wikipedia.org/wiki/Zero-width_joiner
- jahewson 5y ago
  Emoji has turned out to be a great way to enhance complex script support in much real-world software and they get my full support for that reason too.

josephg 5y ago

Handling unicode can be fine, depending on what you're doing. The hard parts are: - Counting, rendering and collapsing grapheme clusters (like the flag emoji) - Converting between legacy encodings (shiftjis, ko8, etc) and UTF-8 / UTF-16 - Canonicalization If all you need is to deal with utf8 byte buffers, you don't need all that stuff. And your code can stay simple, small and fast. IIRC the rust standard library doesn't bother supporting any of the hard parts in unicode. The only real unicode support in std is utf8 validation for strings. All the complex aspects of unicode are delegated to 3rd party crates. By contrast, nodejs (and web browsers) do all of this. But they implement it in the same way you're suggesting - they simply call out to libicu.

tialaramex 5y ago
> The only real unicode support in std is utf8 validation for strings. Rust's core library gives char methods such as is_numeric which asks whether this Unicode codepoint is in one of Unicode's numeric classes such as the letter-like-numerics and various digits. (Rust does provide char with is_ascii_digit and is_ascii_hexdigit if that's all you actually cared about) So yes, the Rust standard library is carrying around the entire Unicode standard class rule list among other things, of course Rust's library isn't built to a vast binary, so if you never use these features your binary doesn't get that code.

amelius 5y ago

Why is this stuff even reinvented for every programming language? Isn't it about time that we have some common language that every other language builds on?

IncRnd 5y ago
So said every writer of a standard immediately before writing another standard to replace all others.
- rwbhn 5y ago
  Obligatory xkcd https://xkcd.com/927/ https://xkcd.com/927/
lmm 5y ago
ICU is out there with bindings in many languages. People who know what they're doing use ICU.
johndough 5y ago
> Isn't it about time that we have some common language that every other language builds on? That language is C. It is debatable whether it was a good choice, but at least this is how it turned.

DecoPerson 5y ago

Do you have a YouTube for people to subscribe to in anticipation of you releasing your YouTube series about your work? The development processes of new languages is so intriguing.

coreyp_1 5y ago
I'll post about it here on HN after I have a few episodes up.

dagmx 5y ago

It would actually be pretty interesting to see how you use Bison and Flex with utf-8. Most resources say to not bother due to lack of support for Unicode, but they're so ubiquitous

account42 5y ago
Do they need special support for UTF-8? One of the nice things about UTF-8 is that you can treat it as an 8-bit encoding in many cases if you only care about substrings and don't need to decode individual non-ASCII characters.
- dagmx 5y ago
  At some point in the pipeline you need the tooling to constrain identifiers to XID_START and XID_CONTINUE

Vindicis 5y ago

I know how that feels. I wrote a little c++ program to fetch data in Unicode from a dB and then normalize it to ascii to be used for analytic purposes. A lot faster to do it on ascii than trying to handle all the fun cases of how many ways can an e etc... be input. ICU to the rescue! Took a couple weeks of getting up to speed as ICU itself wasn't too bad to figure out. But, you find out very quickly that to use it, you need to have a good understanding of a number of the Unicode technical reports to actually understand how to make use of it. Fun times indeed.