5 ms·

This might be interesting read to those unfamiliar with CJK, but character bloat(?) isn't remotely a recent thing. It's actually at least a couple hundred years

by hnfong 4y ago

This might be interesting read to those unfamiliar with CJK, but character bloat(?) isn't remotely a recent thing. It's actually at least a couple hundred years old. The Kangxi dictionary (1716), an authoritative dictionary of Chinese characters, contains definitions for 47035 characters, even though only a couple thousand are in common use. Quoting from Wikipedia: "The dictionary was the largest of the traditional dictionaries, containing 47,035 characters. Some 40% of them are graphic variants, however, while others are dead, archaic, or found only once. Fewer than a quarter of the characters it contains are now in common use." All of these archaic (or even bogus in some cases) characters found in the dictionary are now part of the Unicode standard, of course :) The unihan database even has a field that shows the page number where the character appears in the Kangxi dictionary. If you're wondering why 65536 characters isn't enough for everyone, the junk in Kangxi dictionary is a significant contribution.

mytailorisrich 4y ago
I think 'character bloat' is simply inherent to the writing system when characters are written by hand (now that perhaps most written communication is digital people can't use characters that are not already supported) Anyone can invent characters whenever they want, and it's only a question of them sticking or not. I think this is also one of the reasons for the Chinese tendency to push for unification and uniformity.
- lazide 4y ago
  When it’s character based instead of alphabet based, I think it’s the equivalent of coming up with a new word in English, which is basically what you’re describing. Sometimes it’s mashing two previously unrelated ‘words’ together (aka the tons of compound characters in Chinese), other times it’s coming up with something completely new. Same rules apply though, if it doesn’t add value worth the trouble (or get mandated by the powers that be), it’ll eventually just die out or be a curiosity. Also, to keep it tech related: RISC = English CISC/VLIW = Chinese?
  - nneonneo 4y ago
    IIUC Old Chinese was a much more “isolating” language, in that words were typically single characters - meaning that to make new words, you typically needed to make new characters. As it evolved through the ages, “compound” words composed of multiple characters became more common. These days, new words are almost always combinations of multiple characters (often 2, occasionally 3-4).
    - lazide 4y ago
      Any idea if it was due to things like the Confucian Official’s exam system (and corresponding increase in prioritization of education)? More complex characters require more education to understand is my guess. Some of the traditional ones are….. obscure, and crazy complex.
      - R0b0t1 4y ago
        I'm not entirely sure what you mean to ask nor am I a Chinese speaker, but I have myself suspected that the massive variety of characters was a side-effect of having a middle class that was differentiated based on their ability to read. You see various in-group signalling systems similar to this in lots of areas. A good historical example is all the strangly specific words for groups of animals. A history I read of this indicated these terms were first found in books sold to nobility, and they were just made up. But you weren't hip if you weren't reading that literature.
        nneonneo 4y ago
        Chinese has a lot of specific characters for various species of birds (https://en.wikipedia.org/wiki/Radical_196 https://en.wikipedia.org/wiki/Radical_196), fish (https://en.wikipedia.org/wiki/Radical_195 https://en.wikipedia.org/wiki/Radical_195), and "bugs" (which includes small reptiles and certain other non-warm-blooded animals) (https://en.wikipedia.org/wiki/Radical_142 https://en.wikipedia.org/wiki/Radical_142). One of my favorite features of the language is the fact that every element in the periodic table gets its own character - and the characters have radicals that indicate their usual state of matter (钅 for metal, 石 for non-metal solids, 氵 for liquids, 气 for gases) - see https://en.wikipedia.org/wiki/Chemical_elements_in_East_Asian_languages https://en.wikipedia.org/wiki/Chemical_elements_in_East_Asia.... There are also lots of specific words for species of trees (https://en.wikipedia.org/wiki/Radical_75 https://en.wikipedia.org/wiki/Radical_75) and other plants (https://en.wikipedia.org/wiki/Radical_140 https://en.wikipedia.org/wiki/Radical_140), but characters under those radicals also include botanical terms, medicines, things made of wood and plants, etc.
- nwallin 4y ago
  I'm not sure I understand. Most European languages go through cycles where letters are added when languages are mixed together followed by periods of redundant letters disappearing. Old English had something like 39 letters. 'th' used to have its own letter: thorn. I think character proliferation in CJK languages are a result of each word having its own character. The proliferation isn't fundamentally a proliferation of characters, it's a proliferation of words, which happens all the time in all languages. But only in certain languages does this proliferation of words result in additional characters being added to the language.
  - thrown_22 4y ago
    > I'm not sure I understand. Most European languages go through cycles where letters are added when languages are mixed together followed by periods of redundant letters disappearing. Old English had something like 39 letters. 'th' used to have its own letter: thorn. There is a fundamental difference between pictographic languages where glyphs have intrinsic meaning, and alphabetic languages where letters reflect sounds. Old English was much better spelled than current English because it didn't have a spelling. People wrote what they heard. The current mess is because we have 5 centuries of bad standards that can render ghoti as fish. I think you'll agree that the digraph ti, as in nation, is just as nonsensical as sh for the same sound and we'd be much better served by a single glyph for both. We in fact have that already: https://en.wikipedia.org/wiki/International_Phonetic_Alphabet https://en.wikipedia.org/wiki/International_Phonetic_Alphabe... English uses somewhere around 45 of those sounds depending on accent. Th for example renders two distinct sounds: ð and θ. þ is not any better than th, apart from brevity, because it also rendered to ð or θ when spoken depending on context. Chinese is of course as much a pictographic language as English is an alphabetic one. A substantial number of glyphs come from combinations of simpler glyphs which have the same sound as the word you're trying to write.
ComodoHacker 4y ago
>Fewer than a quarter of the characters it contains are now in common use 12K characters in common use is equally impressing for me as a non-Asian.
- adastra22 4y ago
  More like 12k characters currently in use at all. Common use characters are a much smaller set than that. (3k or so?)
- ssnistfajen 4y ago
  It's actually way fewer than that IRL. Japan's official list of commonly used Kanji only has 2136 characters. Taiwan's list has 4808, and the PRC's list has 3500 "frequent" characters with another 3000 supplementary "common" ones. Digitization has made it even easier to use these characters without recognizing the actual form or how to write them.
  - cyphar 4y ago
    The 常用漢字 (Japanese Common Use Kanji) list does not include many kanji that native speakers can read and newspapers don't always follow the rule that they only should use characters from the list. In addition, you need to include the 人名用漢字 (Personal Name Use Kanji) in the list because basically all of those characters are also used in fairly common words. Native speakers can probably recognise at least 3-4k kanji if not more but can probably only write around 2k from memory, depending on how well-read they are. 嘘 (lie) is the best example of an incredibly common word whose kanji form (which is used fairly often) is not in any official government list.
  - DiogenesKynikos 4y ago
    If you look at a frequency list of Chinese characters,[0] the top 4800 characters make up about 99.9% of modern texts. That means that if you know 4800 characters, and you read a text that is 1000 characters (equivalent to around 700 words) long, there's likely one character you won't recognize. The funny thing is, if you recognize only the top six characters, you already know 10% of the characters in a typical text. The distribution is very top-heavy, but with a long tail that you do have to learn to become literate. 0. https://lingua.mtsu.edu/chinese-computing/statistics/char/list.php?Which=MO https://lingua.mtsu.edu/chinese-computing/statistics/char/li...
ars 4y ago
Does Unicode really need to store Chinese words? Is it impossible to deconstruct the glyphs into strokes, each stroke effectively being a character?
- j16sdiz 4y ago
  Many attempted, but nobody have suceed. The most famous one is `Chu, B.F.: 漢字基因朱邦復漢字基因工程 (Genetic engineering of Chinese characters) (2003), http://cbflabs.com/down/show.php?id=26 http://cbflabs.com/down/show.php?id=26 `
- yongjik 4y ago
  The problem with that would be that every software must know the intricate rules about combining glyphs, and if they guess wrong, users get garbage characters. Considering that the majority of code is written by people who don't know Chinese characters, it would result in never-ending issues, pretty much everywhere. Korean actually has a two-way system in Unicode. Every conceivable character (= syllable) possible in modern Korean has its own codepoint, which allows most software to display them correctly: from their point of view, it's just another CJK character. On the other hand, there is a Unicode area containing Korean sub-blocks ("jamo") that were used historically. In theory, you can combine them and get some pretty funky archaic syllables. Almost no software renders them right.
  - mike_hock 4y ago
    They can't even get much simpler things right. Qt incorrectly combines accents with the character to the right instead of the left and has been refusing to fix this bug for years.
  - deleted 4y ago
    [deleted]
- nneonneo 4y ago
  Unicode has support for this, in the Ideographic Description Characters block (https://en.m.wikipedia.org/wiki/Ideographic_Description_Characters https://en.m.wikipedia.org/wiki/Ideographic_Description_Char...). However, it’s purely descriptive, and not designed for rendering. There are somewhat more sophisticated systems which define both the rendering and stroke decomposition of characters (e.g. CDL: http://guide.wenlininstitute.org/wenlin4.3/Character_Description_Language http://guide.wenlininstitute.org/wenlin4.3/Character_Descrip...). The general workaround for characters that aren’t on Unicode would be to use one of these stroke description systems to create the character, then render it to an image and insert it.
thrown_22 4y ago
Unicode is a mistake that could only have happened in turn of the century America. It is the distilled essence of the idea that you need to be inclusive of everyone along with a fundamental ignorance of what anyone who isn't American does. The idea that Chinese characters are glyphs in the same sense of Latin characters can only have come from someone who has never written Chinese. It is as stupid as demanding a glyph point for each possible integral, e.g. https://quicklatex.com/cache3/8c/ql_9739884527bd893429657272d6a9138c_l3.png https://quicklatex.com/cache3/8c/ql_9739884527bd893429657272... and https://quicklatex.com/cache3/18/ql_c51509950f58a52253c696a419dbe518_l3.png https://quicklatex.com/cache3/18/ql_c51509950f58a52253c696a4.... Solutions for English are not solutions for all languages. You can tell because the solutions that were natively invented by people who spoke those languages were _not_ unicode.
- cyphar 4y ago
  JIS, Big5, UHC, and GB all use a codepoint-to-character approach. You're right to point out that many aspects of "multilingual" support are written by people who do not know another language and so end up being hopelessly misguided but it's not really fair to say that Unicode invented this and thrust it upon the CJK world. Every pre-existing system of representing 漢字 had a codepoint table (which Unicode references in their description of each character). Han Unification was in my view problematic but was driven by technical limitations (then again, if Simplified Chinese characters had also been unified I suspect there would've been more pushback to come up with a better solution, but ultimately Japanese was stuck with being the only one making a major compromise on that front). I don't think a stroke based or combination system would've been better for many reasons: https://news.ycombinator.com/item?id=32102093 https://news.ycombinator.com/item?id=32102093. And if you don't trust Americans who at least tried to learn about the subject matter, how much do you trust any other programmer (who has no interest in other languages) to be able to handle a more complicated system for representing and rendering 漢字?
  - wodenokoto 4y ago
    > Every pre-existing system of representing 漢字 had a codepoint table (which Unicode references in their description of each character). Even typewriters worked with a giant table: https://en.wikipedia.org/wiki/Chinese_typewriter https://en.wikipedia.org/wiki/Chinese_typewriter

by hnfong 4y ago

mytailorisrich 4y ago

I think 'character bloat' is simply inherent to the writing system when characters are written by hand (now that perhaps most written communication is digital people can't use characters that are not already supported) Anyone can invent characters whenever they want, and it's only a question of them sticking or not. I think this is also one of the reasons for the Chinese tendency to push for unification and uniformity.

lazide 4y ago
When it’s character based instead of alphabet based, I think it’s the equivalent of coming up with a new word in English, which is basically what you’re describing. Sometimes it’s mashing two previously unrelated ‘words’ together (aka the tons of compound characters in Chinese), other times it’s coming up with something completely new. Same rules apply though, if it doesn’t add value worth the trouble (or get mandated by the powers that be), it’ll eventually just die out or be a curiosity. Also, to keep it tech related: RISC = English CISC/VLIW = Chinese?
- nneonneo 4y ago
  IIUC Old Chinese was a much more “isolating” language, in that words were typically single characters - meaning that to make new words, you typically needed to make new characters. As it evolved through the ages, “compound” words composed of multiple characters became more common. These days, new words are almost always combinations of multiple characters (often 2, occasionally 3-4).
  - lazide 4y ago
    Any idea if it was due to things like the Confucian Official’s exam system (and corresponding increase in prioritization of education)? More complex characters require more education to understand is my guess. Some of the traditional ones are….. obscure, and crazy complex.
    - R0b0t1 4y ago
      I'm not entirely sure what you mean to ask nor am I a Chinese speaker, but I have myself suspected that the massive variety of characters was a side-effect of having a middle class that was differentiated based on their ability to read. You see various in-group signalling systems similar to this in lots of areas. A good historical example is all the strangly specific words for groups of animals. A history I read of this indicated these terms were first found in books sold to nobility, and they were just made up. But you weren't hip if you weren't reading that literature.
      - nneonneo 4y ago
        Chinese has a lot of specific characters for various species of birds (https://en.wikipedia.org/wiki/Radical_196 https://en.wikipedia.org/wiki/Radical_196), fish (https://en.wikipedia.org/wiki/Radical_195 https://en.wikipedia.org/wiki/Radical_195), and "bugs" (which includes small reptiles and certain other non-warm-blooded animals) (https://en.wikipedia.org/wiki/Radical_142 https://en.wikipedia.org/wiki/Radical_142). One of my favorite features of the language is the fact that every element in the periodic table gets its own character - and the characters have radicals that indicate their usual state of matter (钅 for metal, 石 for non-metal solids, 氵 for liquids, 气 for gases) - see https://en.wikipedia.org/wiki/Chemical_elements_in_East_Asian_languages https://en.wikipedia.org/wiki/Chemical_elements_in_East_Asia.... There are also lots of specific words for species of trees (https://en.wikipedia.org/wiki/Radical_75 https://en.wikipedia.org/wiki/Radical_75) and other plants (https://en.wikipedia.org/wiki/Radical_140 https://en.wikipedia.org/wiki/Radical_140), but characters under those radicals also include botanical terms, medicines, things made of wood and plants, etc.
nwallin 4y ago
I'm not sure I understand. Most European languages go through cycles where letters are added when languages are mixed together followed by periods of redundant letters disappearing. Old English had something like 39 letters. 'th' used to have its own letter: thorn. I think character proliferation in CJK languages are a result of each word having its own character. The proliferation isn't fundamentally a proliferation of characters, it's a proliferation of words, which happens all the time in all languages. But only in certain languages does this proliferation of words result in additional characters being added to the language.
- thrown_22 4y ago
  > I'm not sure I understand. Most European languages go through cycles where letters are added when languages are mixed together followed by periods of redundant letters disappearing. Old English had something like 39 letters. 'th' used to have its own letter: thorn. There is a fundamental difference between pictographic languages where glyphs have intrinsic meaning, and alphabetic languages where letters reflect sounds. Old English was much better spelled than current English because it didn't have a spelling. People wrote what they heard. The current mess is because we have 5 centuries of bad standards that can render ghoti as fish. I think you'll agree that the digraph ti, as in nation, is just as nonsensical as sh for the same sound and we'd be much better served by a single glyph for both. We in fact have that already: https://en.wikipedia.org/wiki/International_Phonetic_Alphabet https://en.wikipedia.org/wiki/International_Phonetic_Alphabe... English uses somewhere around 45 of those sounds depending on accent. Th for example renders two distinct sounds: ð and θ. þ is not any better than th, apart from brevity, because it also rendered to ð or θ when spoken depending on context. Chinese is of course as much a pictographic language as English is an alphabetic one. A substantial number of glyphs come from combinations of simpler glyphs which have the same sound as the word you're trying to write.

ComodoHacker 4y ago

>Fewer than a quarter of the characters it contains are now in common use 12K characters in common use is equally impressing for me as a non-Asian.

adastra22 4y ago
More like 12k characters currently in use at all. Common use characters are a much smaller set than that. (3k or so?)
ssnistfajen 4y ago
It's actually way fewer than that IRL. Japan's official list of commonly used Kanji only has 2136 characters. Taiwan's list has 4808, and the PRC's list has 3500 "frequent" characters with another 3000 supplementary "common" ones. Digitization has made it even easier to use these characters without recognizing the actual form or how to write them.
- cyphar 4y ago
  The 常用漢字 (Japanese Common Use Kanji) list does not include many kanji that native speakers can read and newspapers don't always follow the rule that they only should use characters from the list. In addition, you need to include the 人名用漢字 (Personal Name Use Kanji) in the list because basically all of those characters are also used in fairly common words. Native speakers can probably recognise at least 3-4k kanji if not more but can probably only write around 2k from memory, depending on how well-read they are. 嘘 (lie) is the best example of an incredibly common word whose kanji form (which is used fairly often) is not in any official government list.
- DiogenesKynikos 4y ago
  If you look at a frequency list of Chinese characters,[0] the top 4800 characters make up about 99.9% of modern texts. That means that if you know 4800 characters, and you read a text that is 1000 characters (equivalent to around 700 words) long, there's likely one character you won't recognize. The funny thing is, if you recognize only the top six characters, you already know 10% of the characters in a typical text. The distribution is very top-heavy, but with a long tail that you do have to learn to become literate. 0. https://lingua.mtsu.edu/chinese-computing/statistics/char/list.php?Which=MO https://lingua.mtsu.edu/chinese-computing/statistics/char/li...

ars 4y ago

Does Unicode really need to store Chinese words? Is it impossible to deconstruct the glyphs into strokes, each stroke effectively being a character?

j16sdiz 4y ago
Many attempted, but nobody have suceed. The most famous one is `Chu, B.F.: 漢字基因朱邦復漢字基因工程 (Genetic engineering of Chinese characters) (2003), http://cbflabs.com/down/show.php?id=26 http://cbflabs.com/down/show.php?id=26 `
yongjik 4y ago
The problem with that would be that every software must know the intricate rules about combining glyphs, and if they guess wrong, users get garbage characters. Considering that the majority of code is written by people who don't know Chinese characters, it would result in never-ending issues, pretty much everywhere. Korean actually has a two-way system in Unicode. Every conceivable character (= syllable) possible in modern Korean has its own codepoint, which allows most software to display them correctly: from their point of view, it's just another CJK character. On the other hand, there is a Unicode area containing Korean sub-blocks ("jamo") that were used historically. In theory, you can combine them and get some pretty funky archaic syllables. Almost no software renders them right.
- mike_hock 4y ago
  They can't even get much simpler things right. Qt incorrectly combines accents with the character to the right instead of the left and has been refusing to fix this bug for years.
- deleted 4y ago
  [deleted]
nneonneo 4y ago
Unicode has support for this, in the Ideographic Description Characters block (https://en.m.wikipedia.org/wiki/Ideographic_Description_Characters https://en.m.wikipedia.org/wiki/Ideographic_Description_Char...). However, it’s purely descriptive, and not designed for rendering. There are somewhat more sophisticated systems which define both the rendering and stroke decomposition of characters (e.g. CDL: http://guide.wenlininstitute.org/wenlin4.3/Character_Description_Language http://guide.wenlininstitute.org/wenlin4.3/Character_Descrip...). The general workaround for characters that aren’t on Unicode would be to use one of these stroke description systems to create the character, then render it to an image and insert it.

thrown_22 4y ago

Unicode is a mistake that could only have happened in turn of the century America. It is the distilled essence of the idea that you need to be inclusive of everyone along with a fundamental ignorance of what anyone who isn't American does. The idea that Chinese characters are glyphs in the same sense of Latin characters can only have come from someone who has never written Chinese. It is as stupid as demanding a glyph point for each possible integral, e.g. https://quicklatex.com/cache3/8c/ql_9739884527bd893429657272d6a9138c_l3.png https://quicklatex.com/cache3/8c/ql_9739884527bd893429657272... and https://quicklatex.com/cache3/18/ql_c51509950f58a52253c696a419dbe518_l3.png https://quicklatex.com/cache3/18/ql_c51509950f58a52253c696a4.... Solutions for English are not solutions for all languages. You can tell because the solutions that were natively invented by people who spoke those languages were _not_ unicode.

cyphar 4y ago
JIS, Big5, UHC, and GB all use a codepoint-to-character approach. You're right to point out that many aspects of "multilingual" support are written by people who do not know another language and so end up being hopelessly misguided but it's not really fair to say that Unicode invented this and thrust it upon the CJK world. Every pre-existing system of representing 漢字 had a codepoint table (which Unicode references in their description of each character). Han Unification was in my view problematic but was driven by technical limitations (then again, if Simplified Chinese characters had also been unified I suspect there would've been more pushback to come up with a better solution, but ultimately Japanese was stuck with being the only one making a major compromise on that front). I don't think a stroke based or combination system would've been better for many reasons: https://news.ycombinator.com/item?id=32102093 https://news.ycombinator.com/item?id=32102093. And if you don't trust Americans who at least tried to learn about the subject matter, how much do you trust any other programmer (who has no interest in other languages) to be able to handle a more complicated system for representing and rendering 漢字?
- wodenokoto 4y ago
  > Every pre-existing system of representing 漢字 had a codepoint table (which Unicode references in their description of each character). Even typewriters worked with a giant table: https://en.wikipedia.org/wiki/Chinese_typewriter https://en.wikipedia.org/wiki/Chinese_typewriter