5 ms·

I'll say it again: this is the consequence of Unicode trying to be a mix of html and docx, instead of a charset. It went too far for an average Joe DevGuy to un

by wruza 2y ago

I'll say it again: this is the consequence of Unicode trying to be a mix of html and docx, instead of a charset. It went too far for an average Joe DevGuy to understand how to deal with it, so he just selects a subset he can handle and bans everything else. HN does that too - special symbols simply get removed. Unicode screwed itself up completely. We wanted a common charset for things like latin, extlatin, cjk, cyrillic, hebrew, etc. And we got it, for a while. Shortly after it focused on becoming a complex file format with colorful icons and invisible symbols, which is not manageable without cutting out all that bs by force.

Muromec 2y ago
>so he just selects a subset he can handle and bans everything else. Yes? And the problem is?
- throwaway290 2y ago
  The next guy with a different subset? :)
  - Muromec 2y ago
    The subset is mostly defined by the jurisdiction you operate in, which usually defines a process to map names from one subset to another and is also in the business of keeping the log of said operation. The problem is not operating in a subset, but defining it wrong and not being aware there are multiple of those. If different parts of your system operate in different jurisdictions (or interface which other systems that do), you have to pick multiple subsets and ask user to provide input for each of them. You just can't put anything other than ASCII into either payment card or PNR and the rules of minimal length will differ for the two and you can't put ASCII into the government database which explicitly rejects all of ASCII letters.
    - throwaway290 2y ago
      HN does not accept emoji because of jurisdiction huh?
      - Muromec 2y ago
        That depends on what political philosophy you follow -- they either do, or are wrong and mean.
        throwaway290 2y ago
        I was being sarcastic. As top comment said if Unicode was not a joke and epitomization of feature creep this would be a non issue.
- wruza 2y ago
  The problem is the scale at which it happens and lack of methods-to-go in most runtimes/libs. No one and nothing is ready for unicode complexity out of box, and there's little interest in unscrewing it by oneself, cause it looks like an absurd minefield and likely is one, from the persepective of an average developer. So they get defensive by default, which results in $subj.
mason_mpls 2y ago
This frustration seems unnecessary, unicode isnt more complicated than time and we have far more than enough processing power to handle its most absurd manifestations. We just need good libraries, which is a lot less work than inventing yet another system.
- arka2147483647 2y ago
  The limiting factor is not compute power, but the time and understanding of a random dev somewhere. Time also is not well understood by most programmers. Most just seem to convert it to epoch and pretend that it is continuous.
n2d4 2y ago
> and invisible symbols Invisible symbols were in Unicode before Unicode was even a thing (ASCII already has a few). I also don't think emojis are the reason why devs add checks like in the OP, it's much more likely that they just don't want to deal with character encoding hell. As much as devs like to hate on emojis, they're widely adopted in the real world. Emojis are the closest thing we have to a universal language. Having them in the character encoding standard ensures that they are really universal, and supported by every platform; a loss for everyone who's trying to count the number of glyphs in a string, but a win for everyone else.
- account42 2y ago
  > Emojis are the closest thing we have to a universal language. What meaning does U+1F52B have? What about U+1F346? A set of glyphs does not make a language.
meew0 2y ago
The “invisible symbols” are necessary to correctly represent human language. For instance, one of the most infamous Unicode control characters — the right-to-left override — is required to correctly encode mixed Latin and Hebrew text [1], which are both scripts that you mentioned. Besides, ASCII has control characters as well. The “colorful icons” are not part of Unicode. Emoji are just characters like any other. There is a convention that applications should display them as little coloured images, but this convention has evolved on its own. If you say that Unicode is too expansive, you would have to make a decision to exclude certain types of human communication from being encodable. In my opinion, including everything without discrimination is much preferable here. [1]: https://en.wikipedia.org/wiki/Right-to-left_mark#Example_of_use_in_HTML https://en.wikipedia.org/wiki/Right-to-left_mark#Example_of_...
- account42 2y ago
  > The “colorful icons” are not part of Unicode. Emoji are just characters like any other. There is a convention that applications should display them as little coloured images, but this convention has evolved on its own. Ok now you're just full of shit and you know it.
- n2d4 2y ago
  Granted, technically speaking emojis are not part of the "Unicode Standard", but they are standardized by the Unicode Consortium and constitute "Unicode Technical Standard #51": https://www.unicode.org/reports/tr51/ https://www.unicode.org/reports/tr51/
- Y_Y 2y ago
  I'm happy to discriminate against those damn ancient Sumerians and anyone still using goddamn Linear B.
  - Analemma_ 2y ago
    Sure, but removing those wouldn't make Unicode any simpler, they're just character sets. The GP is complaining about things like combining characters and diacritic modifiers, which make Unicode "ugly" but are necessary if you want to represent real languages used by billions of people.
virexene 2y ago
in what way is unicode similar to html, docx, or a file format? the only features I can think of that are even remotely similar to what you're describing are emoji modifiers. and no, this webpage is not result of "carefully cutting out the complicated stuff from Unicode". i'm pretty sure it's just the result of not supporting Unicode in any meaningful way.
asddubs 2y ago
>We wanted a common charset for things like latin, extlatin, cjk, cyrillic, hebrew, etc. And we got it, for a while. we didn't even get that because slightly different looking characters from japanese and chinese (and other languages) got merged to be the same character in unicode due to having the same origin, meaning you have to use a font based on the language context for it to display correctly.
- tadfisher 2y ago
  They are the same character, though. They do not use the same glyph in different language contexts, but Unicode is a character encoding, not a font standard.
  - Muromec 2y ago
    Yes, but the same is true for overlapping characters in Cyrillic and Latin. A and А are the same glyph, so are т,к,і and t,k,i and you can even see the difference between some of those.
    - tadfisher 2y ago
      The duplication there is mostly to remain compatible or trivially transformable with existing encodings. Ironically, the two versions of your example "A" do look different on my device (Android), with a slightly lower x-height for the Cyrillic version.
      - numpad0 2y ago
        The irony is you calling it irony. CJK "the same or trivially derived" characters are nowhere close to that yet given same code points. CJK unified ideographs is just broken.
  - numpad0 2y ago
    They're not. Readers native in one version can't read the other, and there are more than handful that got duplicated in multiple forms, so they're just not same, just similar. You know, obvious presumption underlying Han Unification is that CJK languages must have a continuous dialect continuums, like villagers living in the middle of East China Sea between Shanghai and Nagasaki and Gwangju would speak half-Chinese-Japanese-Korean, and technical distinction only exist because of rivalry or something. Alas, people don't really erect a house on the surface of an ocean, and CJK languages are each complete isolates with no known shared ancestries, so "it's gotta be all the same" thinking really don't work. I know it's not very intuitive to think that Chinese and Japanese has ZERO syntactic similarity or mutual intelligibility despite relatively tiny mental shares they occupy, but it's just how things are.
throwaway290 2y ago
I bet the complex file format thing probably started at CJK. They wanted to compose Hangul and later someone had a bright idea to do the same to change the look of emojis. Don't worry, AI is the new hotness. All they need is unpack prompts into arbitrary images and finally unicode is truly unicode, all our problems will be solved forever
kristopolous 2y ago
There's no argument here. We could say it's only for script and alphabets, ok. It includes many undeciphered writing systems from antiquity with only a small handful of extent samples. Should we keep that, very likely to never be used character set, but exclude the extremely popular emojis? Exclude both? Why? Aren't computers capable enough? I used to be on the anti emoji bandwagon but really, it's all indefensible. Unicode is characters of communication at an extremely inclusive level. I'm sure some day it will also have primitive shapes and you can construct your own alphabet using them + directional modifiers akin to a generalizable Hangul in effect becoming some kind of wacky version of svg that people will abuse it in an ASCII art renaissance. So be it. Sounds great.
- riwsky 2y ago
  Like how phonetic alphabets save space compared to ideograms by just “write the word how it sounds”, the little SVG-icode would just “write the letter how it’s drawn”
  - deleted 2y ago
    [deleted]
  - kristopolous 2y ago
    Right. Semantic iconography need not be universal or even formal to be real. Think of all the symbols notetakers invent; ideographs without even phonology assigned to it. Being as dynamic as flexible as human expression is hard. Emojis have even taken on this property naturally. The high-5 is also the praying hands for instance. Culturally specific semantics are assigned to the variety of shapes, such as the eggplant and peach. Insisting that this shouldn't happen is a losing battle against how humans construct written language. Good luck with that.
- simonh 2y ago
  No, no, no, no, no… So then we’d get ‘the same’ character with potentially infinite different encodings. Lovely. Unicode is a coding system, not a glyph system or font.
bawolff 2y ago
There are no emoiji in this guy's name. Unicode has made some mistakes, but having all the symbols necessary for this guy's name is not one of them.
jrochkind1 2y ago
Unicode has metadata on each character that would allow software to easily strip out or normalize emoji's and "decorative" characters. It might have edge case problems -- but the charcters in the OP's name would not be included. Also, stripping out emoji's may not actually be required or the right solution. If security is the concern, Unicode also has recommended processes and algorithms for dealing with that. https://www.unicode.org/reports/tr39/ https://www.unicode.org/reports/tr39/ We need better support for the functions developers actually need on unicode in more platforms and languages. Global human language is complicated as a domain. Legacy issues in actually existing data adds to the complexity. Unicode does a pretty good job at it. It's actually pretty amazing how well it does. Including a lot more than just the character set, and encoding, but algorithms for various kinds of normalizing, sorting, indexing, under various localizations, etc. It needs better support in the environments more developers are working in, with raised-to-the-top standard solutions for identified common use cases and problems, that can be implemented simply by calling a performance-optimized library function. (And, if we really want to argue about emoji's, they seem to be extremely popular, and literally have effected global culture, because people want to use them? Blaming emoji's seems like blaming the user! Unicode's support for them actually supports interoperability and vendor-neutral standards for a thing that is wildly popular? but I actually don't think any of the problems or complexity we are talking about, including the OP's complaint, can or should be laid at the feet of emojis)
- deleted 2y ago
  [deleted]
numpad0 2y ago
IMO, the sin of Unicode is that they didn't just pick local language authorities and gave them standardized concepts like lines and characters, and start-of-language and end-of-language markers. Lots of Unicode issues come from handling languages that the code is not expecting, and codes currently has no means to select or report quirk supports. I suppose they didn't like getting national borders involved in technical standardization bit that's just unavoidable. It is getting involved anyway, and these problems are popping up anyway.
- kmeisthax 2y ago
  This doesn't self-synchronize. Removing an arbitrary byte from the text stream (e.g. SOL / EOL) will change the meaning of codepoints far away from the site of the corruption. What it sounds like you want is an easy way for English-language programmers to skip or strip non-ASCII text without having to reference any actual Unicode documentation. Which is a Unicode non-goal, obviously. And also very bad software engineering practice. I'm also not sure what you're getting at with national borders and language authorities, but both of those were absolutely involved with Unicode and still are.
- kevin_thibedeau 2y ago
  > start-of-language and end-of-language markers Unicode used to have language tagging but they've been (mostly) deprecated: https://en.wikipedia.org/wiki/Tags_(Unicode_block) https://en.wikipedia.org/wiki/Tags_(Unicode_block) https://www.unicode.org/reports/tr7/tr7-1.html https://www.unicode.org/reports/tr7/tr7-1.html
  - anonymoushn 2y ago
    The lack of such markers prevents Unicode from encoding strings of mixed Japanese and Chinese text correctly. Or in the case of a piece of software that must accept both Chinese and Japanese names for different people, Unicode is insufficient for encoding the written forms of the names.
    - numpad0 2y ago
jojobas 2y ago
But hey, multiocular o! https://en.wikipedia.org/wiki/Cyrillic_O_variants#Multiocular_O https://en.wikipedia.org/wiki/Cyrillic_O_variants#Multiocula... (TL;DR a bored scribe's doodle has a code point)
- account42 2y ago
  Huh, wasn't aware of this update. > The character was proposed for inclusion into Unicode in 2007 and incorporated as character U+A66E in Unicode version 5.1 (2008). The representative glyph had seven eyes and sat on the baseline. However, in 2021, following a tweet highlighting the character, it came to linguist Michael Everson's attention that the character in the 1429 manuscript was actually made up of ten eyes. After a 2022 proposal to change the character to reflect this, it was updated later that year for Unicode 15.0 to have ten eyes and to extend below the baseline. However, not all fonts support the ten-eyed variant as of November 2024. So not only did they arbitrarily add a non-character (while arbitrarily combinding other real characters) but they didn't even get the glyph right and then changed it to actually match the arbitrary doodle.

by wruza 2y ago

Muromec 2y ago

>so he just selects a subset he can handle and bans everything else. Yes? And the problem is?

throwaway290 2y ago
The next guy with a different subset? :)
- Muromec 2y ago
  The subset is mostly defined by the jurisdiction you operate in, which usually defines a process to map names from one subset to another and is also in the business of keeping the log of said operation. The problem is not operating in a subset, but defining it wrong and not being aware there are multiple of those. If different parts of your system operate in different jurisdictions (or interface which other systems that do), you have to pick multiple subsets and ask user to provide input for each of them. You just can't put anything other than ASCII into either payment card or PNR and the rules of minimal length will differ for the two and you can't put ASCII into the government database which explicitly rejects all of ASCII letters.
  - throwaway290 2y ago
    HN does not accept emoji because of jurisdiction huh?
    - Muromec 2y ago
      That depends on what political philosophy you follow -- they either do, or are wrong and mean.
      - throwaway290 2y ago
        I was being sarcastic. As top comment said if Unicode was not a joke and epitomization of feature creep this would be a non issue.
wruza 2y ago
The problem is the scale at which it happens and lack of methods-to-go in most runtimes/libs. No one and nothing is ready for unicode complexity out of box, and there's little interest in unscrewing it by oneself, cause it looks like an absurd minefield and likely is one, from the persepective of an average developer. So they get defensive by default, which results in $subj.

mason_mpls 2y ago

This frustration seems unnecessary, unicode isnt more complicated than time and we have far more than enough processing power to handle its most absurd manifestations. We just need good libraries, which is a lot less work than inventing yet another system.

arka2147483647 2y ago
The limiting factor is not compute power, but the time and understanding of a random dev somewhere. Time also is not well understood by most programmers. Most just seem to convert it to epoch and pretend that it is continuous.

n2d4 2y ago

> and invisible symbols Invisible symbols were in Unicode before Unicode was even a thing (ASCII already has a few). I also don't think emojis are the reason why devs add checks like in the OP, it's much more likely that they just don't want to deal with character encoding hell. As much as devs like to hate on emojis, they're widely adopted in the real world. Emojis are the closest thing we have to a universal language. Having them in the character encoding standard ensures that they are really universal, and supported by every platform; a loss for everyone who's trying to count the number of glyphs in a string, but a win for everyone else.

account42 2y ago
> Emojis are the closest thing we have to a universal language. What meaning does U+1F52B have? What about U+1F346? A set of glyphs does not make a language.

meew0 2y ago

The “invisible symbols” are necessary to correctly represent human language. For instance, one of the most infamous Unicode control characters — the right-to-left override — is required to correctly encode mixed Latin and Hebrew text [1], which are both scripts that you mentioned. Besides, ASCII has control characters as well. The “colorful icons” are not part of Unicode. Emoji are just characters like any other. There is a convention that applications should display them as little coloured images, but this convention has evolved on its own. If you say that Unicode is too expansive, you would have to make a decision to exclude certain types of human communication from being encodable. In my opinion, including everything without discrimination is much preferable here. [1]: https://en.wikipedia.org/wiki/Right-to-left_mark#Example_of_use_in_HTML https://en.wikipedia.org/wiki/Right-to-left_mark#Example_of_...

account42 2y ago
> The “colorful icons” are not part of Unicode. Emoji are just characters like any other. There is a convention that applications should display them as little coloured images, but this convention has evolved on its own. Ok now you're just full of shit and you know it.
n2d4 2y ago
Granted, technically speaking emojis are not part of the "Unicode Standard", but they are standardized by the Unicode Consortium and constitute "Unicode Technical Standard #51": https://www.unicode.org/reports/tr51/ https://www.unicode.org/reports/tr51/
Y_Y 2y ago
I'm happy to discriminate against those damn ancient Sumerians and anyone still using goddamn Linear B.
- Analemma_ 2y ago
  Sure, but removing those wouldn't make Unicode any simpler, they're just character sets. The GP is complaining about things like combining characters and diacritic modifiers, which make Unicode "ugly" but are necessary if you want to represent real languages used by billions of people.

virexene 2y ago

in what way is unicode similar to html, docx, or a file format? the only features I can think of that are even remotely similar to what you're describing are emoji modifiers. and no, this webpage is not result of "carefully cutting out the complicated stuff from Unicode". i'm pretty sure it's just the result of not supporting Unicode in any meaningful way.

asddubs 2y ago

>We wanted a common charset for things like latin, extlatin, cjk, cyrillic, hebrew, etc. And we got it, for a while. we didn't even get that because slightly different looking characters from japanese and chinese (and other languages) got merged to be the same character in unicode due to having the same origin, meaning you have to use a font based on the language context for it to display correctly.

tadfisher 2y ago
They are the same character, though. They do not use the same glyph in different language contexts, but Unicode is a character encoding, not a font standard.
- Muromec 2y ago
  Yes, but the same is true for overlapping characters in Cyrillic and Latin. A and А are the same glyph, so are т,к,і and t,k,i and you can even see the difference between some of those.
  - tadfisher 2y ago
    The duplication there is mostly to remain compatible or trivially transformable with existing encodings. Ironically, the two versions of your example "A" do look different on my device (Android), with a slightly lower x-height for the Cyrillic version.
    - numpad0 2y ago
      The irony is you calling it irony. CJK "the same or trivially derived" characters are nowhere close to that yet given same code points. CJK unified ideographs is just broken.
- numpad0 2y ago
  They're not. Readers native in one version can't read the other, and there are more than handful that got duplicated in multiple forms, so they're just not same, just similar. You know, obvious presumption underlying Han Unification is that CJK languages must have a continuous dialect continuums, like villagers living in the middle of East China Sea between Shanghai and Nagasaki and Gwangju would speak half-Chinese-Japanese-Korean, and technical distinction only exist because of rivalry or something. Alas, people don't really erect a house on the surface of an ocean, and CJK languages are each complete isolates with no known shared ancestries, so "it's gotta be all the same" thinking really don't work. I know it's not very intuitive to think that Chinese and Japanese has ZERO syntactic similarity or mutual intelligibility despite relatively tiny mental shares they occupy, but it's just how things are.

throwaway290 2y ago

I bet the complex file format thing probably started at CJK. They wanted to compose Hangul and later someone had a bright idea to do the same to change the look of emojis. Don't worry, AI is the new hotness. All they need is unpack prompts into arbitrary images and finally unicode is truly unicode, all our problems will be solved forever

kristopolous 2y ago

There's no argument here. We could say it's only for script and alphabets, ok. It includes many undeciphered writing systems from antiquity with only a small handful of extent samples. Should we keep that, very likely to never be used character set, but exclude the extremely popular emojis? Exclude both? Why? Aren't computers capable enough? I used to be on the anti emoji bandwagon but really, it's all indefensible. Unicode is characters of communication at an extremely inclusive level. I'm sure some day it will also have primitive shapes and you can construct your own alphabet using them + directional modifiers akin to a generalizable Hangul in effect becoming some kind of wacky version of svg that people will abuse it in an ASCII art renaissance. So be it. Sounds great.

riwsky 2y ago
Like how phonetic alphabets save space compared to ideograms by just “write the word how it sounds”, the little SVG-icode would just “write the letter how it’s drawn”
- deleted 2y ago
  [deleted]
- kristopolous 2y ago
  Right. Semantic iconography need not be universal or even formal to be real. Think of all the symbols notetakers invent; ideographs without even phonology assigned to it. Being as dynamic as flexible as human expression is hard. Emojis have even taken on this property naturally. The high-5 is also the praying hands for instance. Culturally specific semantics are assigned to the variety of shapes, such as the eggplant and peach. Insisting that this shouldn't happen is a losing battle against how humans construct written language. Good luck with that.
simonh 2y ago
No, no, no, no, no… So then we’d get ‘the same’ character with potentially infinite different encodings. Lovely. Unicode is a coding system, not a glyph system or font.

bawolff 2y ago

There are no emoiji in this guy's name. Unicode has made some mistakes, but having all the symbols necessary for this guy's name is not one of them.

jrochkind1 2y ago

Unicode has metadata on each character that would allow software to easily strip out or normalize emoji's and "decorative" characters. It might have edge case problems -- but the charcters in the OP's name would not be included. Also, stripping out emoji's may not actually be required or the right solution. If security is the concern, Unicode also has recommended processes and algorithms for dealing with that. https://www.unicode.org/reports/tr39/ https://www.unicode.org/reports/tr39/ We need better support for the functions developers actually need on unicode in more platforms and languages. Global human language is complicated as a domain. Legacy issues in actually existing data adds to the complexity. Unicode does a pretty good job at it. It's actually pretty amazing how well it does. Including a lot more than just the character set, and encoding, but algorithms for various kinds of normalizing, sorting, indexing, under various localizations, etc. It needs better support in the environments more developers are working in, with raised-to-the-top standard solutions for identified common use cases and problems, that can be implemented simply by calling a performance-optimized library function. (And, if we really want to argue about emoji's, they seem to be extremely popular, and literally have effected global culture, because people want to use them? Blaming emoji's seems like blaming the user! Unicode's support for them actually supports interoperability and vendor-neutral standards for a thing that is wildly popular? but I actually don't think any of the problems or complexity we are talking about, including the OP's complaint, can or should be laid at the feet of emojis)

deleted 2y ago
[deleted]

numpad0 2y ago

IMO, the sin of Unicode is that they didn't just pick local language authorities and gave them standardized concepts like lines and characters, and start-of-language and end-of-language markers. Lots of Unicode issues come from handling languages that the code is not expecting, and codes currently has no means to select or report quirk supports. I suppose they didn't like getting national borders involved in technical standardization bit that's just unavoidable. It is getting involved anyway, and these problems are popping up anyway.

kmeisthax 2y ago
This doesn't self-synchronize. Removing an arbitrary byte from the text stream (e.g. SOL / EOL) will change the meaning of codepoints far away from the site of the corruption. What it sounds like you want is an easy way for English-language programmers to skip or strip non-ASCII text without having to reference any actual Unicode documentation. Which is a Unicode non-goal, obviously. And also very bad software engineering practice. I'm also not sure what you're getting at with national borders and language authorities, but both of those were absolutely involved with Unicode and still are.
kevin_thibedeau 2y ago
> start-of-language and end-of-language markers Unicode used to have language tagging but they've been (mostly) deprecated: https://en.wikipedia.org/wiki/Tags_(Unicode_block) https://en.wikipedia.org/wiki/Tags_(Unicode_block) https://www.unicode.org/reports/tr7/tr7-1.html https://www.unicode.org/reports/tr7/tr7-1.html
- anonymoushn 2y ago
  The lack of such markers prevents Unicode from encoding strings of mixed Japanese and Chinese text correctly. Or in the case of a piece of software that must accept both Chinese and Japanese names for different people, Unicode is insufficient for encoding the written forms of the names.
  - numpad0 2y ago

jojobas 2y ago

But hey, multiocular o! https://en.wikipedia.org/wiki/Cyrillic_O_variants#Multiocular_O https://en.wikipedia.org/wiki/Cyrillic_O_variants#Multiocula... (TL;DR a bored scribe's doodle has a code point)

account42 2y ago
Huh, wasn't aware of this update. > The character was proposed for inclusion into Unicode in 2007 and incorporated as character U+A66E in Unicode version 5.1 (2008). The representative glyph had seven eyes and sat on the baseline. However, in 2021, following a tweet highlighting the character, it came to linguist Michael Everson's attention that the character in the 1429 manuscript was actually made up of ten eyes. After a 2022 proposal to change the character to reflect this, it was updated later that year for Unicode 15.0 to have ten eyes and to extend below the baseline. However, not all fonts support the ten-eyed variant as of November 2024. So not only did they arbitrarily add a non-character (while arbitrarily combinding other real characters) but they didn't even get the glyph right and then changed it to actually match the arbitrary doodle.