7 ms·

Are there any Unicode experts here? I've been exchanging emails with Richard Cook (the Unihan maintainer) about getting some "rare" Taiwanese characters added

Are there any Unicode experts here? I've been exchanging emails with Richard Cook (the Unihan maintainer) about getting some "rare" Taiwanese characters added to Unicode. I say "rare" because they're in the Bible, which I think should be covered as a basic text (it's the most-read book in the world!) My research into word spacing, font issues, and more is covered on the blog at https://pingtype.github.io https://pingtype.github.io (click the Docs or Blog header). Practical suggestions for better web design are also welcome. Regarding fonts, this is my specific rant that made me move from Heiti to Pingfang. Unfortunately forcing users to download 13 MB of Pingfang font was too slow for mobile, so I decided to disable it for the web version of Pingtype. https://pingtype.github.io/docs/glyphErrors.html https://pingtype.github.io/docs/glyphErrors.html Edit: These are the IDS codes of the missing characters. Photo evidence from a paper Bible: https://www.flickr.com/photos/150180606@N08/sets/72157693398863955 https://www.flickr.com/photos/150180606@N08/sets/72157693398... 　 ⿱髟煮 chhang.jpg Job 39:19, Job 4:15  ⿸疒粒not𤷟 liap.jpg 1Sa 5:6, 1Sa 5:9, 1Sa 5:12 ... (17 found) - also see WikiSource.  ⿱⿳亠口冖足 37106亮足 lo-.jpg Deu 1:28, Deu 2:10, Deu 9:2 　 ⿰牜周 tiau.jpg 1Ch 17:7, 1Sa 24:3, 2Ch 14:15 (25 found, although 2Ch 14:15 uses 牧 in the paper version)

ChrisSD 9y ago
I'm curious, how popular is the bible in Taiwan? And why does it use Taiwanese characters not commonly used elsewhere?
- oh_sigh 9y ago
  About 5% of Taiwanese are Christian, but culturally Christianity has played a bigger role than you might expect based on that number.
  - yorwba 9y ago
    If the Taiping Rebellion had been successful, China might have become a (very unorthodox, not accepted by any Western church) Christian theocracy. https://en.wikipedia.org/wiki/Taiping_Rebellion https://en.wikipedia.org/wiki/Taiping_Rebellion
    - meric 9y ago
      One quarter as many deaths as World War 2, all of it in southern China.
      - B1FF_PSUVM 9y ago
        When I first learned of it, years ago, I was stunned at two things - the scale of loss of life, and how little known it was ...
    - pavlov 9y ago
      The short description of Taiping's policies [1] makes them sound like a Christian Taliban, or the society of Margaret Atwood's "A Handmaid's Tale". [1] https://en.wikipedia.org/wiki/Taiping_Rebellion#Taiping_Heavenly_Kingdom's_policies https://en.wikipedia.org/wiki/Taiping_Rebellion#Taiping_Heav...
- dwohnitmok 9y ago
  I don't know about the former, but I can answer the latter (and the presumably implied question of why doesn't Unicode cover what should be a fairly common use case?). This Bible is not written in Mandarin, but rather Hokkien/Min (I suppose it's more correct to say "Southern Min," but I'm used to saying Hokkien and you'll see Taiwanese Hokkien as a term appear). And this is actually a kind of interesting statement about the written language. If you've ever talked about languages/dialects in the Chinese language family you might have heard something like "They might sound different but they all share the same underlying written language and are hence dialects" or the opposite "They're entirely separate languages akin to the Romance languages with their own spoken and written languages that might occasionally be similar/have cognates across different languages." The situation is a bit muddier than either of those statements might suggest. As this example demonstrates, it's certainly NOT true that if you just wrote down any language/dialect in the Chinese language family it's intelligible to a Mandarin speaker. Chinese Unicode caters towards Mandarin and as you can see there's certain characters here that essentially never show up in Mandarin (and hence a Mandarin speaker might not recognize them). There are also characters that a Mandarin speaker might recognize but not understand its meaning in this case (e.g. some of the pronouns for example the third-person pronoun 伊 which is normally 他/她/它 or 其 in standard Mandarin depending on how formal the setting is). And certainly any Mandarin speaker can attest to the fact that there are certain languages/dialects that are not verbally mutually intelligible. However, there often exists a written "embedding" of Mandarin into these dialects/languages that doesn't really have an analog among e.g. the Romance languages. For example, Cantonese pop music is often written in Mandarin, but sung in Cantonese, which means that each Mandarin character has a proper Cantonese pronunciation. It comes off as "Mandanrinized" and it's not the way people normally speak, but there's a standard, well-defined way of doing this, and it's intelligible (well at least in certain settings; if you speak like that normally you're going to have a problem). There isn't really a similar way of doing this for French and Spanish. You can't just take a Spanish work, pronounce it using a French pronunciation, and then voila have a French work. It's not even entirely clear for all words what "pronouncing it with a French pronunciation" means. Nonetheless it's not the same thing as having a shared written language. Standard written Cantonese is going to be tricky for a Mandarin reader to read, just as this Hokkien Bible is going to be tricky for a Mandarin reader to read. And even this "embedding" I'm talking about isn't perfect since there are some Chinese characters that show up commonly in Mandarin but rarely in, say, Hokkien and vice versa. Having a shared set of characters only gets you so far when they can have somewhat divergent meanings and very divergent pronunciations. And it's not even the case that all dialects/languages have standardized written forms. Regardless, despite the fact that there are many named languages/dialects in the Chinese language family, the mutual intelligibility situation isn't quite as fragmented as it's occasionally made out to be. There's a handful of large families that share a pretty high degree of mutual intelligibility among dialects/languages within them. A few examples are the Mandarin family of dialects (of which Standard Mandarin is a member), which probably has the broadest geographical distribution, running mainly in a diagonal from the Southwest of China (Sichuan) up to the Northeast (HeilongJiang), the Cantonese family of dialects, the Wu family of dialects, and the Min family of dialects. Returning to this Bible, the funny thing here is that the preface to this Bible is written in Mandarin, but the Bible itself is written in Hokkien (which hints at some of the weird interconnectedness among Chinese dialects/languages). In turn the Chinese characters here are a transliteration of a previous Romanization of an Amoy (Xiamenese Min) translation of the Bible. This post ended up running away from me a bit, so just to recap the original Unicode answer: Chinese Unicode caters towards characters that show up in written Mandarin and this isn't written Mandarin.
- jhanschoo 9y ago
  Something to think about is that in many languages and dialects, the Bible is frequently one of the premier pieces of vernacular literature written in the register that it is spoken. In the case of already-literate societies, until modern nationalism, typically what they write is a classical version of what they speak, which they regard as not prestigious enough to put to pen.
devy 9y ago
No offense and not to sound political here, but there isn't a such language as "Taiwanese". People in Taiwan write in Traditional Chinese characters and use Mandarine (official spoken language and the majority) and Southern Min (or Hokkien proper)[1] as their spoken language. Granted they might have invented a few characters here or there and had variations in some pronunciations in words, but if you ask a linguist expert in Sino-Tibetan languages, they will probably tell you the same thing. [1] https://en.wikipedia.org/wiki/Southern_Min https://en.wikipedia.org/wiki/Southern_Min
- jessriedel 9y ago
  He didn't claim the language was Taiwanese, only the characters. That seems reasonable for Chinese characters that were used predominantly in Taiwan, no?
  - dwohnitmok 9y ago
    This is a somewhat confusing way of putting it. This Bible isn't written in Mandarin (PRC or Taiwanese dialects thereof) and that's why it's using these non-Unihan characters. A Taiwanese Mandarin version wouldn't be using these characters. This Bible is written in (Taiwanese) Hokkien and that's why the GP is having issues with finding these characters in Unihan. EDIT: I do appreciate that "Taiwanese" as in Taiwanese characters is ambiguous. I initially also read that as a "characters used in works written for a Taiwanese audience in Taiwanese Mandarin."
  - kristianov 9y ago
    The characters are all Chinese. Same characters can be found in written Cantonese dialect of Chinese.
- subway 9y ago
  GP never even implied there was a separate Taiwanese language, only that the characters were Taiwanese.
  - ptaipale 9y ago
spacehunt 9y ago
As a native speaker (well, reader), I think I agree with Unicode's opinion that these are stylistic differences only. To me it's like you are trying to argue that the single-storey lowercase "a"[1] is wrong and every font must be in the double-storey style. [1] https://en.wikipedia.org/wiki/A#Typographic_variants https://en.wikipedia.org/wiki/A#Typographic_variants
- git-pull 9y ago
  Since we're on the subject, I'd like to take the time to mention UNIHAN, and also an effort I've created to make historical / regional variants of CJK characters more easily accessible. UNIHAN is Unicode's Han Unification effort. It handles the variant issue - but actually also goes a step further, citing information from paper books, including stroke information, definitions, and even pronunciations [1]. I've compiled a overview of UNIHAN at https://unihan-etl.git-pull.com/en/latest/unihan.html https://unihan-etl.git-pull.com/en/latest/unihan.html. unihan-etl is a project I've created that allows extracting the contents' of UNIHAN's database: https://unihan-etl.git-pull.com https://unihan-etl.git-pull.com. It can be used as a Python library, or a self-serve export of the database to a tabular or structured format. In addition, there is something I've worked on to make this data also available in SQLAlchemy / DB form: https://unihan-db.git-pull.com https://unihan-db.git-pull.com And also using it as a basis for a spiritual successor to cjklib [2]: https://cihai.git-pull.com https://cihai.git-pull.com [1] https://www.unicode.org/reports/tr38/ https://www.unicode.org/reports/tr38/ [2] https://github.com/cburgmer/cjklib https://github.com/cburgmer/cjklib
jhanschoo 9y ago
> Are there any Unicode experts here? Have you considered sending Ken Lunde (@ken_lunde) a tweet? He's quite the expert in CJKV and he usually responds promptly to a tweet. The example regarding wang4 seems like something that should be handled in terms of a Simplified/Traditional variant switch. On the other hand, the unusual characters in the Taiwanese Bible seem to be characters without meaning outside of Taiwanese (or more generally, Min), but that have specific semantic meaning in Taiwanese, and should be encoded in Unicode.
kristianov 9y ago
There is no Taiwanese. Just Chinese.
- intro-b 9y ago
  there are plenty of regional and area-specific dialects of traditional chinese that would require different glyphs
- jhanschoo 9y ago
  By Taiwanese, parent comment was referring to Taiwanese Hokkien, which is mutually unintelligible with Standard Mandarin. Even in writing, Taiwanese written at its spoken register is at best read with great difficulty by Mandarin speakers.
  - kristianov 9y ago
    Get your facts straight. Hokkien, or Minnan Proper (閩南語/閩南話), is a Southern Min dialect group of Chinese. Ask anyone in Fujian Province.
    - jhanschoo 9y ago
      I don't think we are in contradiction. Taiwanese Hokkien is a variety of Min, which is a dialect that shares ancient roots with the Mandarin dialect. My maternal grandmother speaks Teochew, an offshoot of the more common standard Hokkien in the area where I live. Unlike Mandarin, most dialects have not undergone the same nationalist push to write their spoken language, so literature in the spoken language is few and far between. However, Christian proselytizers are often very determined to translate the Bible into local spoken languages, and are often one of the first significant works in any spoken language/dialect's literature. Hence I am not surprised that it is the Bible that top-commenter found digitally-disadvantaged Chinese characters not encountered in literature outside of Taiwanese Hokkien. You may be right in a sense that these same characters may be encountered in other Min dialects, but they would still be pronounced differently.
  - seanmcdirmid 9y ago

ChrisSD 9y ago

I'm curious, how popular is the bible in Taiwan? And why does it use Taiwanese characters not commonly used elsewhere?

oh_sigh 9y ago
About 5% of Taiwanese are Christian, but culturally Christianity has played a bigger role than you might expect based on that number.
- yorwba 9y ago
  If the Taiping Rebellion had been successful, China might have become a (very unorthodox, not accepted by any Western church) Christian theocracy. https://en.wikipedia.org/wiki/Taiping_Rebellion https://en.wikipedia.org/wiki/Taiping_Rebellion
  - meric 9y ago
    One quarter as many deaths as World War 2, all of it in southern China.
    - B1FF_PSUVM 9y ago
      When I first learned of it, years ago, I was stunned at two things - the scale of loss of life, and how little known it was ...
  - pavlov 9y ago
    The short description of Taiping's policies [1] makes them sound like a Christian Taliban, or the society of Margaret Atwood's "A Handmaid's Tale". [1] https://en.wikipedia.org/wiki/Taiping_Rebellion#Taiping_Heavenly_Kingdom's_policies https://en.wikipedia.org/wiki/Taiping_Rebellion#Taiping_Heav...
dwohnitmok 9y ago
I don't know about the former, but I can answer the latter (and the presumably implied question of why doesn't Unicode cover what should be a fairly common use case?). This Bible is not written in Mandarin, but rather Hokkien/Min (I suppose it's more correct to say "Southern Min," but I'm used to saying Hokkien and you'll see Taiwanese Hokkien as a term appear). And this is actually a kind of interesting statement about the written language. If you've ever talked about languages/dialects in the Chinese language family you might have heard something like "They might sound different but they all share the same underlying written language and are hence dialects" or the opposite "They're entirely separate languages akin to the Romance languages with their own spoken and written languages that might occasionally be similar/have cognates across different languages." The situation is a bit muddier than either of those statements might suggest. As this example demonstrates, it's certainly NOT true that if you just wrote down any language/dialect in the Chinese language family it's intelligible to a Mandarin speaker. Chinese Unicode caters towards Mandarin and as you can see there's certain characters here that essentially never show up in Mandarin (and hence a Mandarin speaker might not recognize them). There are also characters that a Mandarin speaker might recognize but not understand its meaning in this case (e.g. some of the pronouns for example the third-person pronoun 伊 which is normally 他/她/它 or 其 in standard Mandarin depending on how formal the setting is). And certainly any Mandarin speaker can attest to the fact that there are certain languages/dialects that are not verbally mutually intelligible. However, there often exists a written "embedding" of Mandarin into these dialects/languages that doesn't really have an analog among e.g. the Romance languages. For example, Cantonese pop music is often written in Mandarin, but sung in Cantonese, which means that each Mandarin character has a proper Cantonese pronunciation. It comes off as "Mandanrinized" and it's not the way people normally speak, but there's a standard, well-defined way of doing this, and it's intelligible (well at least in certain settings; if you speak like that normally you're going to have a problem). There isn't really a similar way of doing this for French and Spanish. You can't just take a Spanish work, pronounce it using a French pronunciation, and then voila have a French work. It's not even entirely clear for all words what "pronouncing it with a French pronunciation" means. Nonetheless it's not the same thing as having a shared written language. Standard written Cantonese is going to be tricky for a Mandarin reader to read, just as this Hokkien Bible is going to be tricky for a Mandarin reader to read. And even this "embedding" I'm talking about isn't perfect since there are some Chinese characters that show up commonly in Mandarin but rarely in, say, Hokkien and vice versa. Having a shared set of characters only gets you so far when they can have somewhat divergent meanings and very divergent pronunciations. And it's not even the case that all dialects/languages have standardized written forms. Regardless, despite the fact that there are many named languages/dialects in the Chinese language family, the mutual intelligibility situation isn't quite as fragmented as it's occasionally made out to be. There's a handful of large families that share a pretty high degree of mutual intelligibility among dialects/languages within them. A few examples are the Mandarin family of dialects (of which Standard Mandarin is a member), which probably has the broadest geographical distribution, running mainly in a diagonal from the Southwest of China (Sichuan) up to the Northeast (HeilongJiang), the Cantonese family of dialects, the Wu family of dialects, and the Min family of dialects. Returning to this Bible, the funny thing here is that the preface to this Bible is written in Mandarin, but the Bible itself is written in Hokkien (which hints at some of the weird interconnectedness among Chinese dialects/languages). In turn the Chinese characters here are a transliteration of a previous Romanization of an Amoy (Xiamenese Min) translation of the Bible. This post ended up running away from me a bit, so just to recap the original Unicode answer: Chinese Unicode caters towards characters that show up in written Mandarin and this isn't written Mandarin.
jhanschoo 9y ago
Something to think about is that in many languages and dialects, the Bible is frequently one of the premier pieces of vernacular literature written in the register that it is spoken. In the case of already-literate societies, until modern nationalism, typically what they write is a classical version of what they speak, which they regard as not prestigious enough to put to pen.

devy 9y ago

No offense and not to sound political here, but there isn't a such language as "Taiwanese". People in Taiwan write in Traditional Chinese characters and use Mandarine (official spoken language and the majority) and Southern Min (or Hokkien proper)[1] as their spoken language. Granted they might have invented a few characters here or there and had variations in some pronunciations in words, but if you ask a linguist expert in Sino-Tibetan languages, they will probably tell you the same thing. [1] https://en.wikipedia.org/wiki/Southern_Min https://en.wikipedia.org/wiki/Southern_Min

jessriedel 9y ago
He didn't claim the language was Taiwanese, only the characters. That seems reasonable for Chinese characters that were used predominantly in Taiwan, no?
- dwohnitmok 9y ago
  This is a somewhat confusing way of putting it. This Bible isn't written in Mandarin (PRC or Taiwanese dialects thereof) and that's why it's using these non-Unihan characters. A Taiwanese Mandarin version wouldn't be using these characters. This Bible is written in (Taiwanese) Hokkien and that's why the GP is having issues with finding these characters in Unihan. EDIT: I do appreciate that "Taiwanese" as in Taiwanese characters is ambiguous. I initially also read that as a "characters used in works written for a Taiwanese audience in Taiwanese Mandarin."
- kristianov 9y ago
  The characters are all Chinese. Same characters can be found in written Cantonese dialect of Chinese.
subway 9y ago
GP never even implied there was a separate Taiwanese language, only that the characters were Taiwanese.
- ptaipale 9y ago

spacehunt 9y ago

As a native speaker (well, reader), I think I agree with Unicode's opinion that these are stylistic differences only. To me it's like you are trying to argue that the single-storey lowercase "a"[1] is wrong and every font must be in the double-storey style. [1] https://en.wikipedia.org/wiki/A#Typographic_variants https://en.wikipedia.org/wiki/A#Typographic_variants

git-pull 9y ago
Since we're on the subject, I'd like to take the time to mention UNIHAN, and also an effort I've created to make historical / regional variants of CJK characters more easily accessible. UNIHAN is Unicode's Han Unification effort. It handles the variant issue - but actually also goes a step further, citing information from paper books, including stroke information, definitions, and even pronunciations [1]. I've compiled a overview of UNIHAN at https://unihan-etl.git-pull.com/en/latest/unihan.html https://unihan-etl.git-pull.com/en/latest/unihan.html. unihan-etl is a project I've created that allows extracting the contents' of UNIHAN's database: https://unihan-etl.git-pull.com https://unihan-etl.git-pull.com. It can be used as a Python library, or a self-serve export of the database to a tabular or structured format. In addition, there is something I've worked on to make this data also available in SQLAlchemy / DB form: https://unihan-db.git-pull.com https://unihan-db.git-pull.com And also using it as a basis for a spiritual successor to cjklib [2]: https://cihai.git-pull.com https://cihai.git-pull.com [1] https://www.unicode.org/reports/tr38/ https://www.unicode.org/reports/tr38/ [2] https://github.com/cburgmer/cjklib https://github.com/cburgmer/cjklib

jhanschoo 9y ago

> Are there any Unicode experts here? Have you considered sending Ken Lunde (@ken_lunde) a tweet? He's quite the expert in CJKV and he usually responds promptly to a tweet. The example regarding wang4 seems like something that should be handled in terms of a Simplified/Traditional variant switch. On the other hand, the unusual characters in the Taiwanese Bible seem to be characters without meaning outside of Taiwanese (or more generally, Min), but that have specific semantic meaning in Taiwanese, and should be encoded in Unicode.

kristianov 9y ago

There is no Taiwanese. Just Chinese.

intro-b 9y ago
there are plenty of regional and area-specific dialects of traditional chinese that would require different glyphs
jhanschoo 9y ago
By Taiwanese, parent comment was referring to Taiwanese Hokkien, which is mutually unintelligible with Standard Mandarin. Even in writing, Taiwanese written at its spoken register is at best read with great difficulty by Mandarin speakers.
- kristianov 9y ago
  Get your facts straight. Hokkien, or Minnan Proper (閩南語/閩南話), is a Southern Min dialect group of Chinese. Ask anyone in Fujian Province.
  - jhanschoo 9y ago
    I don't think we are in contradiction. Taiwanese Hokkien is a variety of Min, which is a dialect that shares ancient roots with the Mandarin dialect. My maternal grandmother speaks Teochew, an offshoot of the more common standard Hokkien in the area where I live. Unlike Mandarin, most dialects have not undergone the same nationalist push to write their spoken language, so literature in the spoken language is few and far between. However, Christian proselytizers are often very determined to translate the Bible into local spoken languages, and are often one of the first significant works in any spoken language/dialect's literature. Hence I am not surprised that it is the Bible that top-commenter found digitally-disadvantaged Chinese characters not encountered in literature outside of Taiwanese Hokkien. You may be right in a sense that these same characters may be encountered in other Min dialects, but they would still be pronounced differently.
- seanmcdirmid 9y ago