13 ms·

A Spectre Is Haunting Unicode (2018)

268 points by EvanAnderson 4y ago

jmillikin 4y ago
Previously: https://news.ycombinator.com/item?id=24951130 https://news.ycombinator.com/item?id=24951130 (2020) https://news.ycombinator.com/item?id=17637375 https://news.ycombinator.com/item?id=17637375 (2018)
krossitalk 4y ago
> At this rate they'll presumably be with humanity forever. Ψ So, that's a really interesting thought. Perhaps our solution to a permanent reminder of nuclear destruction[1] could be hidden inside a plane of Unicode. [1] https://en.wikipedia.org/wiki/Long-term_nuclear_waste_warning_messages https://en.wikipedia.org/wiki/Long-term_nuclear_waste_warnin...
- wongarsu 4y ago
  Maybe we can encode instructions on how to restart society in Unicode character names? After all basically every computer contains a list of them.
  - bogwog 4y ago
    That did not end well for the Georgia guidestones…
    - ethbr0 4y ago
      Also thought about posting that this morning, but wasn't sure anyone else would get the reference. (As context for everyone else, some kook blew up some of the guidestones last week in the middle of the night)
      - _dain_ 4y ago
        Not so kooky, they were a call for genocide.
        tomcatfish 4y ago
        From what I see, this was the maximally flame-y way to say what you said, and it's still inaccurate to call it "Not so kooky" as these commandments, while disagreeable to me, are not really that violent. 1. Maintain humanity under 500,000,000 in perpetual balance with nature. 2. Guide reproduction wisely – improving fitness and diversity. 3. Unite humanity with a living new language. 4. Rule passion – faith – tradition – and all things with tempered reason. 5. Protect people and nations with fair laws and just courts. 6. Let all nations rule internally resolving external disputes in a world court. 7. Avoid petty laws and useless officials. 8. Balance personal rights with social duties. 9. Prize truth – beauty – love – seeking harmony with the infinite. 10. Be not a cancer on the Earth – Leave room for nature – Leave room for nature. source: https://en.wikipedia.org/wiki/Georgia_Guidestones#Inscriptions https://en.wikipedia.org/wiki/Georgia_Guidestones#Inscriptio...
        iamevn 4y ago
        The whole thing stinks of of ecofascism. These first two read as pretty clear calls for eugenicism/genocide to me. > 1. Maintain humanity under 500,000,000 in perpetual balance with nature. > 2. Guide reproduction wisely – improving fitness and diversity.
        astrange 4y ago
        #1 trends towards ecofascism but it's also a major theme in 70s environmentalism (was known as "the population bomb" and is now known as "degrowth") - it's false of course, but many people do believe it. #2 is actually quite unique I think. I mean you could say it's eugenics, but eugenicists are rarely in favor of "diversity".
        inkblotuniverse 4y ago
        Sure they are- some diversity's obviously good so the whole population doesn't get wiped out by one disease. There's a tradeoff between diversity and it's resistance to that sort of disaster, and unity/homogeneity and its resistances to other problems. You'd be a poor eugsnicist to not take that into account!
- remram 4y ago
  Maybe Unicode will feature the same kind of warnings one day. > This Unicode range is not a place of honor. No highly-esteemed symbol is registered here. > What was here represented cultural signs that were considered powerful in our time.
ethbr0 4y ago
The Latin alphabet being boring, I spent some time going through ancient alphabets included in Unicode. It gets pretty trippy, pretty quick. As in "We don't have a clear idea what this rune was for, or what it means, but we see it in documents and so added it to Unicode." https://en.m.wikipedia.org/wiki/Runic_(Unicode_block) https://en.m.wikipedia.org/wiki/Runic_(Unicode_block)
- thaumasiotes 4y ago
  > As in "We don't have a clear idea what this rune was for, or what it means, but we see it in documents and so added it to Unicode." Documents? I had the strong impression that there are no documents written in runes. A rune we only know by its occurrence in documents would be far more interesting for the existence of a document than it would be for its own sake! Compare what the page about Anglo-Saxon runes says about the corpus: > The Old English and Old Frisian Runic Inscriptions database project at the Catholic University of Eichstätt-Ingolstadt, Germany aims at collecting the genuine corpus of Old English inscriptions containing more than two runes in its paper edition, while the electronic edition aims at including both genuine and doubtful inscriptions down to single-rune inscriptions. > The corpus of the paper edition encompasses about one hundred objects (including stone slabs, stone crosses, bones, rings, brooches, weapons, urns, a writing tablet, tweezers, a sun-dial,[clarification needed] comb, bracteates, caskets, a font, dishes, and graffiti). The database includes, in addition, 16 inscriptions containing a single rune, several runic coins, and 8 cases of dubious runic characters (runelike signs, possible Latin characters, weathered characters). Comprising fewer than 200 inscriptions, the corpus is slightly larger than that of Continental Elder Futhark (about 80 inscriptions, c. 400–700), but slightly smaller than that of the Scandinavian Elder Futhark (about 260 inscriptions, c. 200–800). So across every runic system we know, we have under 600 texts, all of those texts are short inscriptions, and even to reach that number of samples we need to include texts that we aren't even sure contain any runes.
  - gumby 4y ago
    > Documents? I had the strong impression that there are no documents written in runes. One of the original goals of Unicode was to be able to computerize every document. I still have some old linguistics books in which characters have been handwritten into typed or even typeset text. So these are the types of documents being referred to: academic papers. Some fancy books have photographs of ancient writing; I’m not sure if Unicode tries to encode such sources and I pretty much doubt it (how would you even know what to call the symbols? You touch on this in your comment). However often they are attached to treatises that order the characters in some way (I.e. index an alphabet) in which case the first case above would apply. In other words: thanks to some scholars who wrote down and ordered runic alphabets, you can now discuss runes with your friends and colleagues through email.
einpoklum 4y ago
I'm more worried about the inflation of emoji than a couple dozen unused ghost JIS characters.
- wongarsu 4y ago
  Godwin's second law: any sufficiently long discussion about Unicode includes a discussion about emoji :)
  - raphlinus 4y ago
    Yeah, it does seem to come up a lot more often than discussions about U+5350.
- edent 4y ago
  Why? Unicode isn't running out of space any time soon.
  - Mountain_Skies 4y ago
    640K should be enough for anybody.
    - rossy 4y ago
      With Unicode, concerns of running out of space are absurd. There are almost a million unallocated codepoints. At the rate codepoints are being allocated, we won't run out for at least 250 years, and emoji are less than 3% of those allocations. Also, the biggest limitation on the number of codepoints is UTF-16, which we have to pray is dead by the late 2200s (when it dies, we'll have over 2 billion unallocated codepoints.)
  - kevin_thibedeau 4y ago
    The encoding has gotten out of hand with compound emoji. Splitting them on glyph boundaries is non-trivial.
helsinkiandrew 4y ago
It looks as if these (at least 妛) are being used in various places on and offline. It’s eventually possible that they will become associated with one or more meanings and perhaps a pronunciation.
- hnfong 4y ago
  In East Asian cultures that use Han characters, people used to make up new characters when the need arises. These days, we scroll though the Unicode standard and find rarely used characters that were accidentally added and imbue them with new meaning. (yes, this is seriously a thing)
  - 1-more 4y ago
    Oooh that sounds fascinating. Any examples of that that spring to mind? Is the pronunciation (or a reasonable representation thereof) already recorded in the Unicode standard or is that also a bit of free-jazz?
    - adastra22 4y ago
      The character usually has a radical component which hints at the pronunciation. They or ordered by radical in the standard. So you would go spelunking for a little-used character in the part of the standard which has characters close in meaning or pronunciation to what you are looking for Or you just make something up. If you’re coining a new character, you probably don’t care about whether the pronunciation is already known.
    - ssnistfajen 4y ago
      An old one but possibly the earliest and most prominent of obsolete Chinese characters being imbued with new (Internet-based) meanings: https://en.wikipedia.org/wiki/Jiong https://en.wikipedia.org/wiki/Jiong There's also 奭 https://en.wiktionary.org/wiki/%E5%A5%AD https://en.wiktionary.org/wiki/%E5%A5%AD which is occasionally used as a censorship workaround to mock one of Xi Jinping's gaffes in an early 2000's TV interview where he bluffed about being able to carry two hundred "catty" (~100kg)'s worth of wheat on rural mountain roads. The character is composed of two 百 ("hundred") and one 人 ("human/person/people") which is a pitoral euphemism to that line he said on TV. I can't find any sources about this one that's in English so please bear with my half-assed explanation.
jamal-kumar 4y ago
I thought this was going to be about something like the massive security problem of homoglyph attacks being currently deployed in stuff like phishing baked into the standard at first glance of the title, but this ghost character business is pretty interesting. Japanese literacy requires you to know 2-4 meanings per 2,136 kanji characters (something like 6000+ in total possible meanings between these characters) just to be able to pass a university level literacy test, it's a massive amount of complexity to get right. Even if you just need basic literacy it's still about a thousand less than that, and there's even more than these I mentioned for further literacy competence. Furthermore each of these characters look funny if not unreadable if you write them down using the wrong order of strokes. I can see how mistakes might have been made even by native speakers of that language. The two kana syllabiaries are there of course and mixed in with the kanji, but if everything was written in that you wouldn't be able to achieve the same amount of information density, which is probably part of the reason they never switched over (I understand before world war 2 or so, the more rounded hiragana was for women while the more sword stroke like katakana was for men).
bqmjjx0kac 4y ago
This is a tangent, but I felt like sharing. In college, I purchased a used copy of the communist manifesto. Famously, the first line reads, "A spectre is haunting Europe, ...". The previous owner had both highlighted and circled the word "spectre" and wrote "ghost?" in the margins. The rest of the text was similarly marked up. Every time I hear the word "spectre" I see "ghost?" in my mind's eye.
jake_morrison 4y ago
In the 90s I worked on a project to digitize land registration in Taiwan. In order to record deeds and property transfers, we needed to enter people's names and official registered addresses into the computer system. The problem was that some people used non-traditional writing variants for their names, and some of their birthplaces were tiny places in China with weird names. Someone might write their name with a two-dot water radical instead of three-dot radical. We would print it out in the normal font, and the people would lose their minds, saying that it was wrong. Chinese people can be superstitious about the number of strokes in their name, so adding a stroke might make it unlucky, so they would not buy the property. The customer went to the agency responsible for managing the big character set, https://en.wikipedia.org/wiki/CNS_11643 https://en.wikipedia.org/wiki/CNS_11643 Despite having more characters than anything else on earth, it didn't have those variants. The agency said they would not encode them, because they were not real characters, just printing differences. The solution was for the staff in the office to use a "font maker" program to create a custom font with these characters. Then they could print out the deeds using a Chinese variant of Adobe Acrobat, and everyone was happy.
- EvanAnderson 4y ago
  Yikes. If somebody hasn’t written a “falsehoods programmers believe about human writing systems” document this would make for a good start.
  - cestith 4y ago
    It deserves its own entry in "falsehoods programmers believe about names" lists too.
    - lmkg 4y ago
      It's already there, #11 "People’s names are all mapped in Unicode code points."
      - Scarblac 4y ago
        The falsehood here is thinking that if you can encode the name into the right code points, and you have a font that can print them, the result will be acceptable to the people whose name it is. They had that, but needed a font that used a different number of strokes for the characters because of the superstition.
hnfong 4y ago
This might be interesting read to those unfamiliar with CJK, but character bloat(?) isn't remotely a recent thing. It's actually at least a couple hundred years old. The Kangxi dictionary (1716), an authoritative dictionary of Chinese characters, contains definitions for 47035 characters, even though only a couple thousand are in common use. Quoting from Wikipedia: "The dictionary was the largest of the traditional dictionaries, containing 47,035 characters. Some 40% of them are graphic variants, however, while others are dead, archaic, or found only once. Fewer than a quarter of the characters it contains are now in common use." All of these archaic (or even bogus in some cases) characters found in the dictionary are now part of the Unicode standard, of course :) The unihan database even has a field that shows the page number where the character appears in the Kangxi dictionary. If you're wondering why 65536 characters isn't enough for everyone, the junk in Kangxi dictionary is a significant contribution.
- mytailorisrich 4y ago
  I think 'character bloat' is simply inherent to the writing system when characters are written by hand (now that perhaps most written communication is digital people can't use characters that are not already supported) Anyone can invent characters whenever they want, and it's only a question of them sticking or not. I think this is also one of the reasons for the Chinese tendency to push for unification and uniformity.
  - lazide 4y ago
    When it’s character based instead of alphabet based, I think it’s the equivalent of coming up with a new word in English, which is basically what you’re describing. Sometimes it’s mashing two previously unrelated ‘words’ together (aka the tons of compound characters in Chinese), other times it’s coming up with something completely new. Same rules apply though, if it doesn’t add value worth the trouble (or get mandated by the powers that be), it’ll eventually just die out or be a curiosity. Also, to keep it tech related: RISC = English CISC/VLIW = Chinese?
    - nneonneo 4y ago
NeoTar 4y ago
Is there anything similar for Latin characters? The only circumstance I can imagine is where a Latin character has been erroneously encoded with an unused diacritic, for instance a T with a diaeresis.
- jxy 4y ago
  Something like the letters V and U.
  - lisper 4y ago
    Or double-U (i.e. UU == W)
- wbl 4y ago
  Multilocular o is known only from a single word in a single manuscript.
  - indecisive_user 4y ago
    link to the wiki article. Though this is a variation of a Cyrillic letter, not latin https://en.wikipedia.org/wiki/Multiocular_O https://en.wikipedia.org/wiki/Multiocular_O
    - asveikau 4y ago
      This is funny. The Indo European (and hence Slavic) roots for eyes typically have an /o/, and this glyph is round like the eye, so it seems like this character and others linked in that article are just people making little cartoonish drawings on writings involving descriptions of eyes.
irusensei 4y ago
If you type 彁 on google translate and set it to detect language it will switch to Chinese and translate it to "lingering". If you switch to Japanese no translation will happen. Also if you google search for 彁 one of the results will be this video [!!!!seizure warning!!!!] https://www.youtube.com/watch?v=EsOU0V2kpUI https://www.youtube.com/watch?v=EsOU0V2kpUI that seems to borrow on the theme of a computer ghost character.
- TazeTSchnitzel 4y ago
  Google Translate will hallucinate translations for complete nonsense, so this probably doesn't mean anything.
  - astrange 4y ago
    WWWJDIC/JMdict also claims it's a name "Junko", but it also isn't very reliable. If you want to know what a Japanese word means you should look it up in a JP-JP dictionary.
olivierestsage 4y ago
Reminds me of the case of U+237C ⍼ RIGHT ANGLE WITH DOWNWARDS ZIGZAG ARROW [0], also discussed on HN [1]. [0] https://ionathan.ch/2022/04/09/angzarr.html https://ionathan.ch/2022/04/09/angzarr.html [1] https://news.ycombinator.com/item?id=31012865 https://news.ycombinator.com/item?id=31012865
lapetitejort 4y ago
I can't be the only person who thought the character would be , right? (based on the first line of the Communist Manifesto: https://en.wikisource.org/wiki/Manifesto_of_the_Communist_Party https://en.wikisource.org/wiki/Manifesto_of_the_Communist_Pa...) edit: ah the character (hammer and sickle) does not show up
ChrisArchitect 4y ago
(2018)
sbf501 4y ago
Can we talk about the artwork used? https://dl.ndl.go.jp/info:ndljp/pid/1312837?itemId=info%3Andljp%2Fpid%2F1312837&__lang=en https://dl.ndl.go.jp/info:ndljp/pid/1312837?itemId=info%3And... https://philamuseum.org/collection/object/84871 https://philamuseum.org/collection/object/84871 Googling for Tsukioka Yoshitoshi brings up so much SEO that it is hard to find information in English. If anyone knows anything about it, I'd be appreciative for a pointer about its content/subject!
- polm23 4y ago
  Author here. Nobody has ever asked about the art before. It depicts Maruyama Oukyo, a famous painter of ghosts (and other things), where one of his pieces comes to life and frightens him. https://en.wikipedia.org/wiki/Maruyama_%C5%8Ckyo https://en.wikipedia.org/wiki/Maruyama_%C5%8Ckyo
js8 4y ago
Is it possible for Unicode standard to deprecate characters? If yes, has it already happened?
- Fell 4y ago
  I don't think so. It would make it impossible to talk about deprecated characters ever again, even in a historical context. Unicode contains even some ancient and long forgotten scripts so historians can keep proper records of them.
- jfk13 4y ago
  Yes, and yes. https://en.wikipedia.org/wiki/Unicode_character_property#Deprecated https://en.wikipedia.org/wiki/Unicode_character_property#Dep...
aatharuv 4y ago
For obviously fake characters, a Unicode proposal for the Egyptian Hieroglyphics Extended-A block managed to include a hieroglyph for an ancient Egyptian holding a laptop. (Note that this is a proposal, and has not yet made it into the standard.) Presumably it was a copyright trap. https://www.unicode.org/mail-arch/unicode-ml/y2020-m02/0018.html https://www.unicode.org/mail-arch/unicode-ml/y2020-m02/0018....
mr_toad 4y ago
Z̵̘̋̎̕ả̶͓͑l̶̜̈͒g̸̡̧̤̋͆õ̶̡͔̥̓ ̵̱͌̈́͝ẃ̴̫̤͘a̶͈̭̱͒̊i̴̭̪̾͑̕ṫ̵͈͙̻s̵̺͐̅̊ ̶̮̩͒̋
brigandish 4y ago
Many (too many) online forms in Japan are unable to process foreign names, sometimes for reasons as brain dead as not allowing more than a few characters in length. Even a perfect unicode standard wouldn't be able to mitigate the arrogance of a programmer.
MrWiffles 4y ago
Summary: Some Japanese characters that aren't real got accidentally built into the unicode code table. This is NOT related to speculative execution attacks at all. It's just "whoa, these kanji don't mean anything, how the hell are they here?!" Other than that caveat (not security related), this is a fascinating article, especially if you've studied Japanese as a (foreign) language.
deleted 4y ago
[deleted]

268 points by EvanAnderson 4y ago

Previously: https://news.ycombinator.com/item?id=24951130 https://news.ycombinator.com/item?id=24951130 (2020) https://news.ycombinator.com/item?id=17637375 https://news.ycombinator.com/item?id=17637375 (2018)

krossitalk 4y ago

> At this rate they'll presumably be with humanity forever. Ψ So, that's a really interesting thought. Perhaps our solution to a permanent reminder of nuclear destruction[1] could be hidden inside a plane of Unicode. [1] https://en.wikipedia.org/wiki/Long-term_nuclear_waste_warning_messages https://en.wikipedia.org/wiki/Long-term_nuclear_waste_warnin...

wongarsu 4y ago
Maybe we can encode instructions on how to restart society in Unicode character names? After all basically every computer contains a list of them.
- bogwog 4y ago
  That did not end well for the Georgia guidestones…
  - ethbr0 4y ago
    Also thought about posting that this morning, but wasn't sure anyone else would get the reference. (As context for everyone else, some kook blew up some of the guidestones last week in the middle of the night)
    - _dain_ 4y ago
      Not so kooky, they were a call for genocide.
      - tomcatfish 4y ago
        From what I see, this was the maximally flame-y way to say what you said, and it's still inaccurate to call it "Not so kooky" as these commandments, while disagreeable to me, are not really that violent. 1. Maintain humanity under 500,000,000 in perpetual balance with nature. 2. Guide reproduction wisely – improving fitness and diversity. 3. Unite humanity with a living new language. 4. Rule passion – faith – tradition – and all things with tempered reason. 5. Protect people and nations with fair laws and just courts. 6. Let all nations rule internally resolving external disputes in a world court. 7. Avoid petty laws and useless officials. 8. Balance personal rights with social duties. 9. Prize truth – beauty – love – seeking harmony with the infinite. 10. Be not a cancer on the Earth – Leave room for nature – Leave room for nature. source: https://en.wikipedia.org/wiki/Georgia_Guidestones#Inscriptions https://en.wikipedia.org/wiki/Georgia_Guidestones#Inscriptio...
        iamevn 4y ago
        The whole thing stinks of of ecofascism. These first two read as pretty clear calls for eugenicism/genocide to me. > 1. Maintain humanity under 500,000,000 in perpetual balance with nature. > 2. Guide reproduction wisely – improving fitness and diversity.
        astrange 4y ago
        #1 trends towards ecofascism but it's also a major theme in 70s environmentalism (was known as "the population bomb" and is now known as "degrowth") - it's false of course, but many people do believe it. #2 is actually quite unique I think. I mean you could say it's eugenics, but eugenicists are rarely in favor of "diversity".
        inkblotuniverse 4y ago
        Sure they are- some diversity's obviously good so the whole population doesn't get wiped out by one disease. There's a tradeoff between diversity and it's resistance to that sort of disaster, and unity/homogeneity and its resistances to other problems. You'd be a poor eugsnicist to not take that into account!
remram 4y ago
Maybe Unicode will feature the same kind of warnings one day. > This Unicode range is not a place of honor. No highly-esteemed symbol is registered here. > What was here represented cultural signs that were considered powerful in our time.

ethbr0 4y ago

The Latin alphabet being boring, I spent some time going through ancient alphabets included in Unicode. It gets pretty trippy, pretty quick. As in "We don't have a clear idea what this rune was for, or what it means, but we see it in documents and so added it to Unicode." https://en.m.wikipedia.org/wiki/Runic_(Unicode_block) https://en.m.wikipedia.org/wiki/Runic_(Unicode_block)

thaumasiotes 4y ago
> As in "We don't have a clear idea what this rune was for, or what it means, but we see it in documents and so added it to Unicode." Documents? I had the strong impression that there are no documents written in runes. A rune we only know by its occurrence in documents would be far more interesting for the existence of a document than it would be for its own sake! Compare what the page about Anglo-Saxon runes says about the corpus: > The Old English and Old Frisian Runic Inscriptions database project at the Catholic University of Eichstätt-Ingolstadt, Germany aims at collecting the genuine corpus of Old English inscriptions containing more than two runes in its paper edition, while the electronic edition aims at including both genuine and doubtful inscriptions down to single-rune inscriptions. > The corpus of the paper edition encompasses about one hundred objects (including stone slabs, stone crosses, bones, rings, brooches, weapons, urns, a writing tablet, tweezers, a sun-dial,[clarification needed] comb, bracteates, caskets, a font, dishes, and graffiti). The database includes, in addition, 16 inscriptions containing a single rune, several runic coins, and 8 cases of dubious runic characters (runelike signs, possible Latin characters, weathered characters). Comprising fewer than 200 inscriptions, the corpus is slightly larger than that of Continental Elder Futhark (about 80 inscriptions, c. 400–700), but slightly smaller than that of the Scandinavian Elder Futhark (about 260 inscriptions, c. 200–800). So across every runic system we know, we have under 600 texts, all of those texts are short inscriptions, and even to reach that number of samples we need to include texts that we aren't even sure contain any runes.
- gumby 4y ago
  > Documents? I had the strong impression that there are no documents written in runes. One of the original goals of Unicode was to be able to computerize every document. I still have some old linguistics books in which characters have been handwritten into typed or even typeset text. So these are the types of documents being referred to: academic papers. Some fancy books have photographs of ancient writing; I’m not sure if Unicode tries to encode such sources and I pretty much doubt it (how would you even know what to call the symbols? You touch on this in your comment). However often they are attached to treatises that order the characters in some way (I.e. index an alphabet) in which case the first case above would apply. In other words: thanks to some scholars who wrote down and ordered runic alphabets, you can now discuss runes with your friends and colleagues through email.

einpoklum 4y ago

I'm more worried about the inflation of emoji than a couple dozen unused ghost JIS characters.

wongarsu 4y ago
Godwin's second law: any sufficiently long discussion about Unicode includes a discussion about emoji :)
- raphlinus 4y ago
  Yeah, it does seem to come up a lot more often than discussions about U+5350.
edent 4y ago
Why? Unicode isn't running out of space any time soon.
- Mountain_Skies 4y ago
  640K should be enough for anybody.
  - rossy 4y ago
    With Unicode, concerns of running out of space are absurd. There are almost a million unallocated codepoints. At the rate codepoints are being allocated, we won't run out for at least 250 years, and emoji are less than 3% of those allocations. Also, the biggest limitation on the number of codepoints is UTF-16, which we have to pray is dead by the late 2200s (when it dies, we'll have over 2 billion unallocated codepoints.)
- kevin_thibedeau 4y ago
  The encoding has gotten out of hand with compound emoji. Splitting them on glyph boundaries is non-trivial.

helsinkiandrew 4y ago

It looks as if these (at least 妛) are being used in various places on and offline. It’s eventually possible that they will become associated with one or more meanings and perhaps a pronunciation.

hnfong 4y ago
In East Asian cultures that use Han characters, people used to make up new characters when the need arises. These days, we scroll though the Unicode standard and find rarely used characters that were accidentally added and imbue them with new meaning. (yes, this is seriously a thing)
- 1-more 4y ago
  Oooh that sounds fascinating. Any examples of that that spring to mind? Is the pronunciation (or a reasonable representation thereof) already recorded in the Unicode standard or is that also a bit of free-jazz?
  - adastra22 4y ago
    The character usually has a radical component which hints at the pronunciation. They or ordered by radical in the standard. So you would go spelunking for a little-used character in the part of the standard which has characters close in meaning or pronunciation to what you are looking for Or you just make something up. If you’re coining a new character, you probably don’t care about whether the pronunciation is already known.
  - ssnistfajen 4y ago
    An old one but possibly the earliest and most prominent of obsolete Chinese characters being imbued with new (Internet-based) meanings: https://en.wikipedia.org/wiki/Jiong https://en.wikipedia.org/wiki/Jiong There's also 奭 https://en.wiktionary.org/wiki/%E5%A5%AD https://en.wiktionary.org/wiki/%E5%A5%AD which is occasionally used as a censorship workaround to mock one of Xi Jinping's gaffes in an early 2000's TV interview where he bluffed about being able to carry two hundred "catty" (~100kg)'s worth of wheat on rural mountain roads. The character is composed of two 百 ("hundred") and one 人 ("human/person/people") which is a pitoral euphemism to that line he said on TV. I can't find any sources about this one that's in English so please bear with my half-assed explanation.

jamal-kumar 4y ago

I thought this was going to be about something like the massive security problem of homoglyph attacks being currently deployed in stuff like phishing baked into the standard at first glance of the title, but this ghost character business is pretty interesting. Japanese literacy requires you to know 2-4 meanings per 2,136 kanji characters (something like 6000+ in total possible meanings between these characters) just to be able to pass a university level literacy test, it's a massive amount of complexity to get right. Even if you just need basic literacy it's still about a thousand less than that, and there's even more than these I mentioned for further literacy competence. Furthermore each of these characters look funny if not unreadable if you write them down using the wrong order of strokes. I can see how mistakes might have been made even by native speakers of that language. The two kana syllabiaries are there of course and mixed in with the kanji, but if everything was written in that you wouldn't be able to achieve the same amount of information density, which is probably part of the reason they never switched over (I understand before world war 2 or so, the more rounded hiragana was for women while the more sword stroke like katakana was for men).

bqmjjx0kac 4y ago

This is a tangent, but I felt like sharing. In college, I purchased a used copy of the communist manifesto. Famously, the first line reads, "A spectre is haunting Europe, ...". The previous owner had both highlighted and circled the word "spectre" and wrote "ghost?" in the margins. The rest of the text was similarly marked up. Every time I hear the word "spectre" I see "ghost?" in my mind's eye.

jake_morrison 4y ago

In the 90s I worked on a project to digitize land registration in Taiwan. In order to record deeds and property transfers, we needed to enter people's names and official registered addresses into the computer system. The problem was that some people used non-traditional writing variants for their names, and some of their birthplaces were tiny places in China with weird names. Someone might write their name with a two-dot water radical instead of three-dot radical. We would print it out in the normal font, and the people would lose their minds, saying that it was wrong. Chinese people can be superstitious about the number of strokes in their name, so adding a stroke might make it unlucky, so they would not buy the property. The customer went to the agency responsible for managing the big character set, https://en.wikipedia.org/wiki/CNS_11643 https://en.wikipedia.org/wiki/CNS_11643 Despite having more characters than anything else on earth, it didn't have those variants. The agency said they would not encode them, because they were not real characters, just printing differences. The solution was for the staff in the office to use a "font maker" program to create a custom font with these characters. Then they could print out the deeds using a Chinese variant of Adobe Acrobat, and everyone was happy.

EvanAnderson 4y ago
Yikes. If somebody hasn’t written a “falsehoods programmers believe about human writing systems” document this would make for a good start.
- cestith 4y ago
  It deserves its own entry in "falsehoods programmers believe about names" lists too.
  - lmkg 4y ago
    It's already there, #11 "People’s names are all mapped in Unicode code points."
    - Scarblac 4y ago
      The falsehood here is thinking that if you can encode the name into the right code points, and you have a font that can print them, the result will be acceptable to the people whose name it is. They had that, but needed a font that used a different number of strokes for the characters because of the superstition.

hnfong 4y ago

This might be interesting read to those unfamiliar with CJK, but character bloat(?) isn't remotely a recent thing. It's actually at least a couple hundred years old. The Kangxi dictionary (1716), an authoritative dictionary of Chinese characters, contains definitions for 47035 characters, even though only a couple thousand are in common use. Quoting from Wikipedia: "The dictionary was the largest of the traditional dictionaries, containing 47,035 characters. Some 40% of them are graphic variants, however, while others are dead, archaic, or found only once. Fewer than a quarter of the characters it contains are now in common use." All of these archaic (or even bogus in some cases) characters found in the dictionary are now part of the Unicode standard, of course :) The unihan database even has a field that shows the page number where the character appears in the Kangxi dictionary. If you're wondering why 65536 characters isn't enough for everyone, the junk in Kangxi dictionary is a significant contribution.

mytailorisrich 4y ago
I think 'character bloat' is simply inherent to the writing system when characters are written by hand (now that perhaps most written communication is digital people can't use characters that are not already supported) Anyone can invent characters whenever they want, and it's only a question of them sticking or not. I think this is also one of the reasons for the Chinese tendency to push for unification and uniformity.
- lazide 4y ago
  When it’s character based instead of alphabet based, I think it’s the equivalent of coming up with a new word in English, which is basically what you’re describing. Sometimes it’s mashing two previously unrelated ‘words’ together (aka the tons of compound characters in Chinese), other times it’s coming up with something completely new. Same rules apply though, if it doesn’t add value worth the trouble (or get mandated by the powers that be), it’ll eventually just die out or be a curiosity. Also, to keep it tech related: RISC = English CISC/VLIW = Chinese?
  - nneonneo 4y ago

NeoTar 4y ago

Is there anything similar for Latin characters? The only circumstance I can imagine is where a Latin character has been erroneously encoded with an unused diacritic, for instance a T with a diaeresis.

jxy 4y ago
Something like the letters V and U.
- lisper 4y ago
  Or double-U (i.e. UU == W)
wbl 4y ago
Multilocular o is known only from a single word in a single manuscript.
- indecisive_user 4y ago
  link to the wiki article. Though this is a variation of a Cyrillic letter, not latin https://en.wikipedia.org/wiki/Multiocular_O https://en.wikipedia.org/wiki/Multiocular_O
  - asveikau 4y ago
    This is funny. The Indo European (and hence Slavic) roots for eyes typically have an /o/, and this glyph is round like the eye, so it seems like this character and others linked in that article are just people making little cartoonish drawings on writings involving descriptions of eyes.

irusensei 4y ago

If you type 彁 on google translate and set it to detect language it will switch to Chinese and translate it to "lingering". If you switch to Japanese no translation will happen. Also if you google search for 彁 one of the results will be this video [!!!!seizure warning!!!!] https://www.youtube.com/watch?v=EsOU0V2kpUI https://www.youtube.com/watch?v=EsOU0V2kpUI that seems to borrow on the theme of a computer ghost character.

TazeTSchnitzel 4y ago
Google Translate will hallucinate translations for complete nonsense, so this probably doesn't mean anything.
- astrange 4y ago
  WWWJDIC/JMdict also claims it's a name "Junko", but it also isn't very reliable. If you want to know what a Japanese word means you should look it up in a JP-JP dictionary.

olivierestsage 4y ago

Reminds me of the case of U+237C ⍼ RIGHT ANGLE WITH DOWNWARDS ZIGZAG ARROW [0], also discussed on HN [1]. [0] https://ionathan.ch/2022/04/09/angzarr.html https://ionathan.ch/2022/04/09/angzarr.html [1] https://news.ycombinator.com/item?id=31012865 https://news.ycombinator.com/item?id=31012865

lapetitejort 4y ago

I can't be the only person who thought the character would be , right? (based on the first line of the Communist Manifesto: https://en.wikisource.org/wiki/Manifesto_of_the_Communist_Party https://en.wikisource.org/wiki/Manifesto_of_the_Communist_Pa...) edit: ah the character (hammer and sickle) does not show up

ChrisArchitect 4y ago

(2018)

sbf501 4y ago

Can we talk about the artwork used? https://dl.ndl.go.jp/info:ndljp/pid/1312837?itemId=info%3Andljp%2Fpid%2F1312837&__lang=en https://dl.ndl.go.jp/info:ndljp/pid/1312837?itemId=info%3And... https://philamuseum.org/collection/object/84871 https://philamuseum.org/collection/object/84871 Googling for Tsukioka Yoshitoshi brings up so much SEO that it is hard to find information in English. If anyone knows anything about it, I'd be appreciative for a pointer about its content/subject!

polm23 4y ago
Author here. Nobody has ever asked about the art before. It depicts Maruyama Oukyo, a famous painter of ghosts (and other things), where one of his pieces comes to life and frightens him. https://en.wikipedia.org/wiki/Maruyama_%C5%8Ckyo https://en.wikipedia.org/wiki/Maruyama_%C5%8Ckyo

js8 4y ago

Is it possible for Unicode standard to deprecate characters? If yes, has it already happened?

Fell 4y ago
I don't think so. It would make it impossible to talk about deprecated characters ever again, even in a historical context. Unicode contains even some ancient and long forgotten scripts so historians can keep proper records of them.
jfk13 4y ago
Yes, and yes. https://en.wikipedia.org/wiki/Unicode_character_property#Deprecated https://en.wikipedia.org/wiki/Unicode_character_property#Dep...

aatharuv 4y ago

For obviously fake characters, a Unicode proposal for the Egyptian Hieroglyphics Extended-A block managed to include a hieroglyph for an ancient Egyptian holding a laptop. (Note that this is a proposal, and has not yet made it into the standard.) Presumably it was a copyright trap. https://www.unicode.org/mail-arch/unicode-ml/y2020-m02/0018.html https://www.unicode.org/mail-arch/unicode-ml/y2020-m02/0018....

mr_toad 4y ago

Z̵̘̋̎̕ả̶͓͑l̶̜̈͒g̸̡̧̤̋͆õ̶̡͔̥̓ ̵̱͌̈́͝ẃ̴̫̤͘a̶͈̭̱͒̊i̴̭̪̾͑̕ṫ̵͈͙̻s̵̺͐̅̊ ̶̮̩͒̋

brigandish 4y ago

Many (too many) online forms in Japan are unable to process foreign names, sometimes for reasons as brain dead as not allowing more than a few characters in length. Even a perfect unicode standard wouldn't be able to mitigate the arrogance of a programmer.

MrWiffles 4y ago

Summary: Some Japanese characters that aren't real got accidentally built into the unicode code table. This is NOT related to speculative execution attacks at all. It's just "whoa, these kanji don't mean anything, how the hell are they here?!" Other than that caveat (not security related), this is a fascinating article, especially if you've studied Japanese as a (foreign) language.

deleted 4y ago

[deleted]