6 ms·

A spectre is haunting Unicode

282 points by sensanaty 2mo ago

joshdavham 2mo ago
The author Paul McCann (polm) is one of my favourite programmers out there! He’s done awesome work in the Japanese NLP space over the last decade which has really helped me in my language learning projects. He maintains a mecab (Japanese tokenizer) wrapper for Python [0], has a book on Japanese NLP written for English speakers [1] and also worked on Spacy at one point [2]. [0] https://github.com/polm/fugashi https://github.com/polm/fugashi [1] https://www.japanesenlp.com/ https://www.japanesenlp.com/ [2] https://spacy.io/ https://spacy.io/
hnfong 2mo ago
Well, vast swaths of the Kangxi dictionary (which serves as "sources" for probably most of the CJK characters) are such "ghost" characters as described in the article... The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP....
- CodesInChaos 2mo ago
  > apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism Do you mean Han Unification? I.e. that conceptually equivalent characters which are written differently in Japan and China received only a single unicode code-point, and are rendered the Chinese way by default on most computers?
  - hnfong 2mo ago
    Yes. As far as I understood it, the result was an incoherent mess. I think some characters got different code points, while others were merged. And apparently the Japanese complained bitterly over the ones that were merged. If you had read any articles about that, this is probably what you have in mind right now. And I'm also a bit tilted by the ones that had different code points, because when processing CJK text now we have to deal with characters that are (in my native Cantonese) essentially the same, looks similar (to my undiscriminating eyes), yet having different code points so that things like text search sometimes don't work. Of course I'm not "blaming" the Japanese, if anything the simplified vs traditional Chinese thing is much more of a practical problem, and the conflicting code points I deal with on a routine basis are more of a Hong Kong vs Taiwan thing, but I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode (which I think really is some kind of Aristotelian essentialism...) mostly due to vocal objections from the Japanese.
    - pamcake 2mo ago
      > I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode What would that philosophy be about? Sounds apocryphal. Unicode has never done "unification" like that for other languages/scripts? i/ı/i, ö/ø/ø̈/oͤ: Same same, different codepoints. Search and sorting is a mess everywhere. Depending on your locale, ö sorts either after o or after z. Sometimes it's semantically and phonetically equivalent to o wrt search but moreoften not. https://en.wikipedia.org/wiki/%C3%96 https://en.wikipedia.org/wiki/%C3%96
      - yorwba 2mo ago
        Well, Unicode didn't unify 門/门 either, but the appearance of the second glyph changes a lot depending on whether you use a Chinese font or a Japanese one. Meanwhile they did unify all the graphical variants of Latin characters with identical semantics, it's just that everyone is so used to the result that you hardly notice not being able to control whether g has a hook or a looped tail, and nobody expects to copy some German blackletter text into a French document and have it show up visually distinct, without explicitly setting the font.
        Joker_vD 2mo ago
        Yes, and Cyrillic letters also look quite different in Bulgarian and Russian fonts. So what?
        asveikau 2mo ago
        Is that true? Other than things like not seeing a Ы, Э or Ё in Bulgarian, the modern printed fonts look the same to me, though I am not from either country.
        yorwba 2mo ago
        Some letters look different: https://en.wikipedia.org/wiki/Bulgarian_alphabet#/media/File:Special_Cyrillics_BGDPT.svg https://en.wikipedia.org/wiki/Bulgarian_alphabet#/media/File...
  - muvlon 2mo ago
    Meanwhile, Latin a and Cyrillic а, which look identical, have the same origin and are often pronounced identically too, get separate codepoints. CJK unification was not based on any clear preestablished principles but simply an attempt to cling to 16-bit codepoints, Asian languages be damned.
    - esrauch 2mo ago
      Cling to 16 bit codepoints sounds like a legitimate explanation? Latin and Cyrillic didn't would have saved no relevant amount of space. It looks like the CJK unified space is over 20,000 characters, so that's a real technical magnitude distinction compared to Latin and Cyrillic. "Asian languages be damned" seems like a bad faith read, compared to "Java and Windows char is 16 bits and that will never change realistically" (and in fact they still haven't, even in 2026 things which rely on UTF16 instead of UCS2 are still commonly bugged unfortunately)
      - Joker_vD 2mo ago
        We also have at least 15 copies of Latin alphabet in Unicode, all differently styled/fonted (there is chancery cursive, mathematical cursive, the fraktur version, the full-width version, etc.)
        gucci-on-fleek 2mo ago
        Yes, but most of those are intended to be used exclusively for math, where the different styles do have semantic meaning. Even in introductory textbooks, it's pretty common to see the roman, calligraphic, and double-struck variants of a single letter all used with different meanings. Plus, these characters were only added 20+ years later, and in the supplemental planes, not the BMP.
        Joker_vD 2mo ago
        Letterlike Symbols and Fullwidth Forms are in BMP (and I generously consider Enclosed Alphanumerics to not be a copy of a Latin alphabet).
  - throwaway2037 2mo ago
    There is a nice list of example Chinese characters that differ between languages, e.g., Chinese vs Korean vs Japanese, etc: https://en.wikipedia.org/wiki/Han_unification#Examples_of_language-dependent_glyphs https://en.wikipedia.org/wiki/Han_unification#Examples_of_la... And then some examples of non-unified Chinese chars: https://en.wikipedia.org/wiki/Han_unification#Examples_of_some_non-unified_Han_ideographs https://en.wikipedia.org/wiki/Han_unification#Examples_of_so...
- RicoElectrico 2mo ago
  The Han unification probably was about fitting CJK into the 16 bits aka BMP. It was a blunder indeed.
  - knuckleheadsmif 2mo ago
    It is a blunder that Japan could not see that they just copied other characters in to JIS and it’s also why we see duplicated Romaji characters within Unicode. The duplication is frankly bad regardless of number of bits need to encode although with 16 it was indeed a requirement.
- gsnedders 2mo ago
  > The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP.... Note that Han Unification has a history predating Unicode, and the Unicode work very much followed on from that. The most notable is CCCII, developed in Taiwan in the early 80s, and then standardised in various places. That's not to say that it hasn't been controversial (it has!), nor to say that it hasn't caused problems (it has!), but it's also unfair to say that it's just Unicode doing its own special thing.
- zx8080 2mo ago
  What is the rationale to the unifying direction of changes in Unicode?
philipov 2mo ago
"- the spectre of communism. All the powers of old encoding have entered into a holy alliance to exorcise this spectre..."
- schoen 2mo ago
  Well, Communism made it into Unicode (U+262D). I tried to show it here but it gets stripped.
  - eviks 2mo ago
    Figures, it's a capitalist website after all...
    - pocksuppet 2mo ago
      YOU SEEM TO BE ABLE TO ALSO TYPE IN LOWERCASE BUT FOR SOME REASon i can't never mind i pressed caps lock everything is better now
  - somat 2mo ago
    I have it it in my .XCompose as <Multi_key> <c> <c> <c> <p> : "\xe2\x98\xad" #symbol representing proletarian solidarity between agricultural and industrial workers Never used, but I laugh every time I see it there. As a slightly related tangent, The compose key mnemonic interface for rarely used characters is pretty great, beats trying to remember alt codes.
    - mjg59 2mo ago
      Originally added to the upstream compose map in https://gitlab.freedesktop.org/xorg/lib/libx11/-/commit/f052665394f3f0319e93a98f1d5d4ea287e1dd07 https://gitlab.freedesktop.org/xorg/lib/libx11/-/commit/f052..., and replaced in https://gitlab.freedesktop.org/xorg/lib/libx11/-/merge_requests/112 https://gitlab.freedesktop.org/xorg/lib/libx11/-/merge_reque.... Given the people involved in the initial commit, I suspect (but do not have an absolute memory of) it may have been influenced by my ownership of (insert a hammer and sickle symbol here).net.
erjiang 2mo ago
I think there’s evidence found for the origin of “彁” as the result of a poor scan of a newspaper article. Look up “彁新聞” to find some japanese sources about this.
- retrac 2mo ago
  Ghost characters reveal something about the joint semantic and phonetic nature of most Chinese characters. I know 彁 isn't real: but it still insists on a pronunciation: ka in Japanese, (gē or gé Mandarin). And it hints at meaning: bowed weapon, sound, elder brother; but your guess is as good as mine!
  - thaumasiotes 2mo ago
    > I know 彁 isn't real: but it still insists on a pronunciation: ka in Japanese, (gē or gé Mandarin). Huh? How do you pronounce 切? > And it hints at meaning: bowed weapon, sound, elder brother; but your guess is as good as mine! Why is "elder brother" a meaning hint if you've already assumed that 哥 is the phonophore?
    - matvore 2mo ago
      A component can contribute meaning as well as pronunciation. 沌 is pronounced like 屯 and they both have a kind of "gather" "clog" meaning
      - thaumasiotes 2mo ago
        That is not a method of character construction. It could happen, but you would never expect it. Also... there is no "gather" or "clog" sense of 沌? There's a "gather" sense of 屯...
        matvore 2mo ago
        According to https://kotobank.jp/word/%E6%B2%8C-2789579#goog_rewarded https://kotobank.jp/word/%E6%B2%8C-2789579#goog_rewarded 沌 has a meaning "gather, water gathers" There are other meanings to it. I admit I had a hard time finding an example.
sedatk 2mo ago
Fascinating. But, I guess it's better to have superfluous invalid characters than missing real ones.
panzi 2mo ago
Is anyone using these characters now for anything? No youth language or online slang using it?
- decimalenough 2mo ago
  There's some fan fiction that treats the characters as names of ghosts and comes up with back stories and visualizations for them (the article links to a compilation, but it's 403ing for me). Nothing mainstream though.
  - panzi 2mo ago
    It's something! I propose that the first 4 of those characters should in English be called Blinky, Pinky, Inky, and Clyde.
eviks 2mo ago
> The original character (𡚴) was not added to JIS or Unicode until much later and doesn't display on most sites for me Why didn't they simly replace the original bad one? > nine hundred pages. Imagine tracking down a single character without a page reference Not that hard to imagine, OCR existed back then?
- Kye 2mo ago
  OCR was slow and unreliable and was for a very long time.
  - eviks 2mo ago
    "slow" - wasn't like they were pressed for time. It took them almost 20years to even start the investigation! Also not certain you needed that much reliability compared to what was available to match one pic to a quality scan to get a reasonable number of candidates
- gucci-on-fleek 2mo ago
  > Not that hard to imagine, OCR existed back then? How do you train OCR on a character that doesn't exist though? Even these days, OCR often confuses the 52 Latin characters, so I wouldn't expect as good of results with older technology and some 20k CJK characters.
  - Izkata 2mo ago
    (Talking about Japanese here) I don't know if this is how OCR would work here, but they're made of components called radicals. It's kind of like how letters make words, but in a two dimensional layout, and one letter was wrong and a nonexistent word was made. For example 彁 is made of parts of 引 and 歌 (the first that popped into my mind). IIRC there's only like 200-something radicals, though some can be hard to spot due to how they overlap or nest inside each other.
    - cyphar 2mo ago
      I find 鸚哥（いんこ） to be the easiest way to type the right hand side of 彁.
gweinberg 2mo ago
It occurs to me that we can use 彊 to mean "a completely unknown concept that cannot be named". For example if you ask, "when Cthulhu rises from its slumber, what is thefirst thing it will do? Probably it will 彊.
- arcanemachiner 2mo ago
  That's not on my keyboard... Can I just stick with "ineffable"?
  - holdupagain 2mo ago
    Not exactly, it doesn't look like the definition or even connotations of "ineffable" include "unknown".
    - speerer 2mo ago
      It does in the sense that ineffable things are unable to be described in language (conceptually: unknown to language).
    - thaumasiotes 2mo ago
      Anything that is known can be described in language by giving it a name.
      - gweinberg 2mo ago
        Hey, that's today's smbc https://www.smbc-comics.com/comic/r https://www.smbc-comics.com/comic/r The problem is, there's no good reason to think the concept of 彊 in your mind is at all similar to the concept of 彊 in my brain. After all, we (probably) don't pronounce it the same, right?
        thaumasiotes 2mo ago
        So learn what the word means. How did you learn what "concept" meant?
amake 2mo ago
Should probably have "(2008)" in the title
xelxebar 2mo ago
Xu Bing has a book that consists entirely of invented characters: https://en.wikipedia.org/wiki/A_Book_from_the_Sky https://en.wikipedia.org/wiki/A_Book_from_the_Sky
Dwedit 2mo ago
Saw headline, expected branch prediction vulnerability involving Unicode, was surprised at a completely different topic.
pocksuppet 2mo ago
A language where you can make up new characters seems anathema to a universal character encoding.
tengwar2 2mo ago
Before that, there was the case of ÿ and Ÿ in the base IBM character set (see https://en.wikipedia.org/wiki/Ÿ https://en.wikipedia.org/wiki/Ÿ). The former exists in a handful of proper nouns in French and Hungarian (as in about five proper nouns), but only as a y with a diæresis, which would normally be handled with a Unicode modifier symbol. It is also used in a couple of endangered languages in South America, which would not have influenced the character getting in to a PC character set. It is thought that possibly it was supposed to represent the Dutch "ij" character, but apparently that is never written as "ÿ" in print. Ÿ seems to not be used by any language.
- pseudohadamard 2mo ago
  Great, so they added Ÿ but not n̈, making it very difficult to write music reviews. (Not sure how the latter will render in your browser).
  - ComputerGuru 2mo ago
    iOS at least renders it as an n with dieresis; is that how it was intended (I’m unfamiliar with musical notation)? If so, what is so difficult about it? In fact, (semantics aside, from a technical perspective) the preference should always be for modifiers rather than standalone characters because the chances of being supported by the viewer’s font are much greater: it doesn’t need a separate glyph explicitly drawn and add to the font file for the code point. Difficulties in entering it or typing it out should be mitigated with client-side affordances in the UI, shortcuts, etc.
    - YoshiRulz 2mo ago
      I assume they meant https://en.wikipedia.org/wiki/Spinal_Tap_(band) https://en.wikipedia.org/wiki/Spinal_Tap_(band) not anything to do with notation.
    - cryptonector 2mo ago
      > In fact, (semantics aside, from a technical perspective) the preference should always be for modifiers rather than standalone characters [...] Yes. But the font has to be designed to allow this. That means that a) lower-case letters must be small enough to allow "overstrike" with diacritical marks to render correctly, b) diacritical marks must be small enough too, c) if you want capitals to also render correctly then the font must have either a set of smaller capitals, or smaller/higher diacritics, and the renderer must scale the capitals and diacritics to fit, or change line spacing, etc. Also, the 'semantics' for the _human_ reader are the same whether you use pre-composed or decomposed codepoint sequences -- the semantics for the human are about the glyph as rendered and not the details of how that glyph was obtained. And to be super-pedantic (sorry!), what you call 'modifiers' are called combining marks in Unicode, and what you call 'standalone characters' are called precompositions in Unicode. And it's not necessarily true that the rendering will _in practice_ work better with the former than the latter, but in theory absolutely it is, and in practice it almost always is for _browsers_. > Difficulties in entering it or typing it out should be mitigated with client-side affordances in the UI, shortcuts, etc. I really wish Windows would adopt X11-style compose key sequences. Those are incredibly natural for all glyphs that can notionally be constructed via 'overstriking', and historically that is exactly how people did construct those with typewriters. (I don't know, but I suspect that for typesetting it was necessary to have a type for each modified character because having decomposed movable type would probably not have been robust enough.)
cyteeditor 2mo ago
[flagged]
schoen 2mo ago
See also https://en.wikipedia.org/wiki/A_Book_from_the_Sky https://en.wikipedia.org/wiki/A_Book_from_the_Sky (a 1988 art project based on intentionally making up ghost characters by hand and printing a whole book with nonexistent but aesthetically plausible Chinese characters). Amusingly, Wikipedia mentions that over a dozen of them (out of about 4,000) were later found to accidentally exist!
i18nagentai 2mo ago
[flagged]
podocarp 2mo ago
So what is the spectre? I thought this was proposing something new like the manifesto
zahlman 2mo ago
(2018) I've seen YouTube videos on this topic before. > The original character (𡚴) was not added to JIS or Unicode until much later and doesn't display on most sites for me. It's from Unicode version 3.1 (published 2001) so this is surprising.
singularity2001 2mo ago
[flagged]
eproxus 2mo ago
This would make a great tattoo, as a pun on people getting Kanji character tattoos without being able to read Kanji.
reindeer2 2mo ago
[flagged]
layer8 2mo ago
(2018) Major previous discussions: 110 comments: https://news.ycombinator.com/item?id=17637375 https://news.ycombinator.com/item?id=17637375 130 comments: https://news.ycombinator.com/item?id=24951130 https://news.ycombinator.com/item?id=24951130 180 comments: https://news.ycombinator.com/item?id=32095502 https://news.ycombinator.com/item?id=32095502
ComputerGuru 2mo ago
It’s really a complete and total nothing burger. Extra code points were added, might have been an issue when we were trying to cap the total number below needing some arbitrarily fixed number of bytes for convenience, but now that’s no longer the case and they’re just a historical oddity that costs nothing to maintain and certainly don’t “haunt” in the sense of “ keep popping up and causing problems” in any way. I appreciate the article nevertheless, of course, but I do feel that it would probably be more meaningful to someone that has at least a basic understanding of Japanese.

282 points by sensanaty 2mo ago

The author Paul McCann (polm) is one of my favourite programmers out there! He’s done awesome work in the Japanese NLP space over the last decade which has really helped me in my language learning projects. He maintains a mecab (Japanese tokenizer) wrapper for Python [0], has a book on Japanese NLP written for English speakers [1] and also worked on Spacy at one point [2]. [0] https://github.com/polm/fugashi https://github.com/polm/fugashi [1] https://www.japanesenlp.com/ https://www.japanesenlp.com/ [2] https://spacy.io/ https://spacy.io/

hnfong 2mo ago

Well, vast swaths of the Kangxi dictionary (which serves as "sources" for probably most of the CJK characters) are such "ghost" characters as described in the article... The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP....

CodesInChaos 2mo ago
> apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism Do you mean Han Unification? I.e. that conceptually equivalent characters which are written differently in Japan and China received only a single unicode code-point, and are rendered the Chinese way by default on most computers?
- hnfong 2mo ago
  Yes. As far as I understood it, the result was an incoherent mess. I think some characters got different code points, while others were merged. And apparently the Japanese complained bitterly over the ones that were merged. If you had read any articles about that, this is probably what you have in mind right now. And I'm also a bit tilted by the ones that had different code points, because when processing CJK text now we have to deal with characters that are (in my native Cantonese) essentially the same, looks similar (to my undiscriminating eyes), yet having different code points so that things like text search sometimes don't work. Of course I'm not "blaming" the Japanese, if anything the simplified vs traditional Chinese thing is much more of a practical problem, and the conflicting code points I deal with on a routine basis are more of a Hong Kong vs Taiwan thing, but I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode (which I think really is some kind of Aristotelian essentialism...) mostly due to vocal objections from the Japanese.
  - pamcake 2mo ago
    > I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode What would that philosophy be about? Sounds apocryphal. Unicode has never done "unification" like that for other languages/scripts? i/ı/i, ö/ø/ø̈/oͤ: Same same, different codepoints. Search and sorting is a mess everywhere. Depending on your locale, ö sorts either after o or after z. Sometimes it's semantically and phonetically equivalent to o wrt search but moreoften not. https://en.wikipedia.org/wiki/%C3%96 https://en.wikipedia.org/wiki/%C3%96
    - yorwba 2mo ago
      Well, Unicode didn't unify 門/门 either, but the appearance of the second glyph changes a lot depending on whether you use a Chinese font or a Japanese one. Meanwhile they did unify all the graphical variants of Latin characters with identical semantics, it's just that everyone is so used to the result that you hardly notice not being able to control whether g has a hook or a looped tail, and nobody expects to copy some German blackletter text into a French document and have it show up visually distinct, without explicitly setting the font.
      - Joker_vD 2mo ago
        Yes, and Cyrillic letters also look quite different in Bulgarian and Russian fonts. So what?
        asveikau 2mo ago
        Is that true? Other than things like not seeing a Ы, Э or Ё in Bulgarian, the modern printed fonts look the same to me, though I am not from either country.
        yorwba 2mo ago
        Some letters look different: https://en.wikipedia.org/wiki/Bulgarian_alphabet#/media/File:Special_Cyrillics_BGDPT.svg https://en.wikipedia.org/wiki/Bulgarian_alphabet#/media/File...
- muvlon 2mo ago
  Meanwhile, Latin a and Cyrillic а, which look identical, have the same origin and are often pronounced identically too, get separate codepoints. CJK unification was not based on any clear preestablished principles but simply an attempt to cling to 16-bit codepoints, Asian languages be damned.
  - esrauch 2mo ago
    Cling to 16 bit codepoints sounds like a legitimate explanation? Latin and Cyrillic didn't would have saved no relevant amount of space. It looks like the CJK unified space is over 20,000 characters, so that's a real technical magnitude distinction compared to Latin and Cyrillic. "Asian languages be damned" seems like a bad faith read, compared to "Java and Windows char is 16 bits and that will never change realistically" (and in fact they still haven't, even in 2026 things which rely on UTF16 instead of UCS2 are still commonly bugged unfortunately)
    - Joker_vD 2mo ago
      We also have at least 15 copies of Latin alphabet in Unicode, all differently styled/fonted (there is chancery cursive, mathematical cursive, the fraktur version, the full-width version, etc.)
      - gucci-on-fleek 2mo ago
        Yes, but most of those are intended to be used exclusively for math, where the different styles do have semantic meaning. Even in introductory textbooks, it's pretty common to see the roman, calligraphic, and double-struck variants of a single letter all used with different meanings. Plus, these characters were only added 20+ years later, and in the supplemental planes, not the BMP.
        Joker_vD 2mo ago
        Letterlike Symbols and Fullwidth Forms are in BMP (and I generously consider Enclosed Alphanumerics to not be a copy of a Latin alphabet).
- throwaway2037 2mo ago
  There is a nice list of example Chinese characters that differ between languages, e.g., Chinese vs Korean vs Japanese, etc: https://en.wikipedia.org/wiki/Han_unification#Examples_of_language-dependent_glyphs https://en.wikipedia.org/wiki/Han_unification#Examples_of_la... And then some examples of non-unified Chinese chars: https://en.wikipedia.org/wiki/Han_unification#Examples_of_some_non-unified_Han_ideographs https://en.wikipedia.org/wiki/Han_unification#Examples_of_so...
RicoElectrico 2mo ago
The Han unification probably was about fitting CJK into the 16 bits aka BMP. It was a blunder indeed.
- knuckleheadsmif 2mo ago
  It is a blunder that Japan could not see that they just copied other characters in to JIS and it’s also why we see duplicated Romaji characters within Unicode. The duplication is frankly bad regardless of number of bits need to encode although with 16 it was indeed a requirement.
gsnedders 2mo ago
> The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP.... Note that Han Unification has a history predating Unicode, and the Unicode work very much followed on from that. The most notable is CCCII, developed in Taiwan in the early 80s, and then standardised in various places. That's not to say that it hasn't been controversial (it has!), nor to say that it hasn't caused problems (it has!), but it's also unfair to say that it's just Unicode doing its own special thing.
zx8080 2mo ago
What is the rationale to the unifying direction of changes in Unicode?

philipov 2mo ago

"- the spectre of communism. All the powers of old encoding have entered into a holy alliance to exorcise this spectre..."

schoen 2mo ago
Well, Communism made it into Unicode (U+262D). I tried to show it here but it gets stripped.
- eviks 2mo ago
  Figures, it's a capitalist website after all...
  - pocksuppet 2mo ago
    YOU SEEM TO BE ABLE TO ALSO TYPE IN LOWERCASE BUT FOR SOME REASon i can't never mind i pressed caps lock everything is better now
- somat 2mo ago
  I have it it in my .XCompose as <Multi_key> <c> <c> <c> <p> : "\xe2\x98\xad" #symbol representing proletarian solidarity between agricultural and industrial workers Never used, but I laugh every time I see it there. As a slightly related tangent, The compose key mnemonic interface for rarely used characters is pretty great, beats trying to remember alt codes.
  - mjg59 2mo ago
    Originally added to the upstream compose map in https://gitlab.freedesktop.org/xorg/lib/libx11/-/commit/f052665394f3f0319e93a98f1d5d4ea287e1dd07 https://gitlab.freedesktop.org/xorg/lib/libx11/-/commit/f052..., and replaced in https://gitlab.freedesktop.org/xorg/lib/libx11/-/merge_requests/112 https://gitlab.freedesktop.org/xorg/lib/libx11/-/merge_reque.... Given the people involved in the initial commit, I suspect (but do not have an absolute memory of) it may have been influenced by my ownership of (insert a hammer and sickle symbol here).net.

erjiang 2mo ago

I think there’s evidence found for the origin of “彁” as the result of a poor scan of a newspaper article. Look up “彁新聞” to find some japanese sources about this.

retrac 2mo ago
Ghost characters reveal something about the joint semantic and phonetic nature of most Chinese characters. I know 彁 isn't real: but it still insists on a pronunciation: ka in Japanese, (gē or gé Mandarin). And it hints at meaning: bowed weapon, sound, elder brother; but your guess is as good as mine!
- thaumasiotes 2mo ago
  > I know 彁 isn't real: but it still insists on a pronunciation: ka in Japanese, (gē or gé Mandarin). Huh? How do you pronounce 切? > And it hints at meaning: bowed weapon, sound, elder brother; but your guess is as good as mine! Why is "elder brother" a meaning hint if you've already assumed that 哥 is the phonophore?
  - matvore 2mo ago
    A component can contribute meaning as well as pronunciation. 沌 is pronounced like 屯 and they both have a kind of "gather" "clog" meaning
    - thaumasiotes 2mo ago
      That is not a method of character construction. It could happen, but you would never expect it. Also... there is no "gather" or "clog" sense of 沌? There's a "gather" sense of 屯...
      - matvore 2mo ago
        According to https://kotobank.jp/word/%E6%B2%8C-2789579#goog_rewarded https://kotobank.jp/word/%E6%B2%8C-2789579#goog_rewarded 沌 has a meaning "gather, water gathers" There are other meanings to it. I admit I had a hard time finding an example.

sedatk 2mo ago

Fascinating. But, I guess it's better to have superfluous invalid characters than missing real ones.

panzi 2mo ago

Is anyone using these characters now for anything? No youth language or online slang using it?

decimalenough 2mo ago
There's some fan fiction that treats the characters as names of ghosts and comes up with back stories and visualizations for them (the article links to a compilation, but it's 403ing for me). Nothing mainstream though.
- panzi 2mo ago
  It's something! I propose that the first 4 of those characters should in English be called Blinky, Pinky, Inky, and Clyde.

eviks 2mo ago

> The original character (𡚴) was not added to JIS or Unicode until much later and doesn't display on most sites for me Why didn't they simly replace the original bad one? > nine hundred pages. Imagine tracking down a single character without a page reference Not that hard to imagine, OCR existed back then?

Kye 2mo ago
OCR was slow and unreliable and was for a very long time.
- eviks 2mo ago
  "slow" - wasn't like they were pressed for time. It took them almost 20years to even start the investigation! Also not certain you needed that much reliability compared to what was available to match one pic to a quality scan to get a reasonable number of candidates
gucci-on-fleek 2mo ago
> Not that hard to imagine, OCR existed back then? How do you train OCR on a character that doesn't exist though? Even these days, OCR often confuses the 52 Latin characters, so I wouldn't expect as good of results with older technology and some 20k CJK characters.
- Izkata 2mo ago
  (Talking about Japanese here) I don't know if this is how OCR would work here, but they're made of components called radicals. It's kind of like how letters make words, but in a two dimensional layout, and one letter was wrong and a nonexistent word was made. For example 彁 is made of parts of 引 and 歌 (the first that popped into my mind). IIRC there's only like 200-something radicals, though some can be hard to spot due to how they overlap or nest inside each other.
  - cyphar 2mo ago
    I find 鸚哥（いんこ） to be the easiest way to type the right hand side of 彁.

gweinberg 2mo ago

It occurs to me that we can use 彊 to mean "a completely unknown concept that cannot be named". For example if you ask, "when Cthulhu rises from its slumber, what is thefirst thing it will do? Probably it will 彊.

arcanemachiner 2mo ago
That's not on my keyboard... Can I just stick with "ineffable"?
- holdupagain 2mo ago
  Not exactly, it doesn't look like the definition or even connotations of "ineffable" include "unknown".
  - speerer 2mo ago
    It does in the sense that ineffable things are unable to be described in language (conceptually: unknown to language).
  - thaumasiotes 2mo ago
    Anything that is known can be described in language by giving it a name.
    - gweinberg 2mo ago
      Hey, that's today's smbc https://www.smbc-comics.com/comic/r https://www.smbc-comics.com/comic/r The problem is, there's no good reason to think the concept of 彊 in your mind is at all similar to the concept of 彊 in my brain. After all, we (probably) don't pronounce it the same, right?
      - thaumasiotes 2mo ago
        So learn what the word means. How did you learn what "concept" meant?

amake 2mo ago

Should probably have "(2008)" in the title

xelxebar 2mo ago

Xu Bing has a book that consists entirely of invented characters: https://en.wikipedia.org/wiki/A_Book_from_the_Sky https://en.wikipedia.org/wiki/A_Book_from_the_Sky

Dwedit 2mo ago

Saw headline, expected branch prediction vulnerability involving Unicode, was surprised at a completely different topic.

pocksuppet 2mo ago

A language where you can make up new characters seems anathema to a universal character encoding.

tengwar2 2mo ago

Before that, there was the case of ÿ and Ÿ in the base IBM character set (see https://en.wikipedia.org/wiki/Ÿ https://en.wikipedia.org/wiki/Ÿ). The former exists in a handful of proper nouns in French and Hungarian (as in about five proper nouns), but only as a y with a diæresis, which would normally be handled with a Unicode modifier symbol. It is also used in a couple of endangered languages in South America, which would not have influenced the character getting in to a PC character set. It is thought that possibly it was supposed to represent the Dutch "ij" character, but apparently that is never written as "ÿ" in print. Ÿ seems to not be used by any language.

pseudohadamard 2mo ago
Great, so they added Ÿ but not n̈, making it very difficult to write music reviews. (Not sure how the latter will render in your browser).
- ComputerGuru 2mo ago
  iOS at least renders it as an n with dieresis; is that how it was intended (I’m unfamiliar with musical notation)? If so, what is so difficult about it? In fact, (semantics aside, from a technical perspective) the preference should always be for modifiers rather than standalone characters because the chances of being supported by the viewer’s font are much greater: it doesn’t need a separate glyph explicitly drawn and add to the font file for the code point. Difficulties in entering it or typing it out should be mitigated with client-side affordances in the UI, shortcuts, etc.
  - YoshiRulz 2mo ago
    I assume they meant https://en.wikipedia.org/wiki/Spinal_Tap_(band) https://en.wikipedia.org/wiki/Spinal_Tap_(band) not anything to do with notation.
  - cryptonector 2mo ago
    > In fact, (semantics aside, from a technical perspective) the preference should always be for modifiers rather than standalone characters [...] Yes. But the font has to be designed to allow this. That means that a) lower-case letters must be small enough to allow "overstrike" with diacritical marks to render correctly, b) diacritical marks must be small enough too, c) if you want capitals to also render correctly then the font must have either a set of smaller capitals, or smaller/higher diacritics, and the renderer must scale the capitals and diacritics to fit, or change line spacing, etc. Also, the 'semantics' for the _human_ reader are the same whether you use pre-composed or decomposed codepoint sequences -- the semantics for the human are about the glyph as rendered and not the details of how that glyph was obtained. And to be super-pedantic (sorry!), what you call 'modifiers' are called combining marks in Unicode, and what you call 'standalone characters' are called precompositions in Unicode. And it's not necessarily true that the rendering will _in practice_ work better with the former than the latter, but in theory absolutely it is, and in practice it almost always is for _browsers_. > Difficulties in entering it or typing it out should be mitigated with client-side affordances in the UI, shortcuts, etc. I really wish Windows would adopt X11-style compose key sequences. Those are incredibly natural for all glyphs that can notionally be constructed via 'overstriking', and historically that is exactly how people did construct those with typewriters. (I don't know, but I suspect that for typesetting it was necessary to have a type for each modified character because having decomposed movable type would probably not have been robust enough.)

cyteeditor 2mo ago

[flagged]

schoen 2mo ago

See also https://en.wikipedia.org/wiki/A_Book_from_the_Sky https://en.wikipedia.org/wiki/A_Book_from_the_Sky (a 1988 art project based on intentionally making up ghost characters by hand and printing a whole book with nonexistent but aesthetically plausible Chinese characters). Amusingly, Wikipedia mentions that over a dozen of them (out of about 4,000) were later found to accidentally exist!

i18nagentai 2mo ago

[flagged]

podocarp 2mo ago

So what is the spectre? I thought this was proposing something new like the manifesto

zahlman 2mo ago

(2018) I've seen YouTube videos on this topic before. > The original character (𡚴) was not added to JIS or Unicode until much later and doesn't display on most sites for me. It's from Unicode version 3.1 (published 2001) so this is surprising.

singularity2001 2mo ago

[flagged]

eproxus 2mo ago

This would make a great tattoo, as a pun on people getting Kanji character tattoos without being able to read Kanji.

reindeer2 2mo ago

[flagged]

layer8 2mo ago

(2018) Major previous discussions: 110 comments: https://news.ycombinator.com/item?id=17637375 https://news.ycombinator.com/item?id=17637375 130 comments: https://news.ycombinator.com/item?id=24951130 https://news.ycombinator.com/item?id=24951130 180 comments: https://news.ycombinator.com/item?id=32095502 https://news.ycombinator.com/item?id=32095502

ComputerGuru 2mo ago

It’s really a complete and total nothing burger. Extra code points were added, might have been an issue when we were trying to cap the total number below needing some arbitrarily fixed number of bytes for convenience, but now that’s no longer the case and they’re just a historical oddity that costs nothing to maintain and certainly don’t “haunt” in the sense of “ keep popping up and causing problems” in any way. I appreciate the article nevertheless, of course, but I do feel that it would probably be more meaningful to someone that has at least a basic understanding of Japanese.