3 ms·

I'm trying to learn Chinese. I wrote http://pingtype.github.io http://pingtype.github.io to parse blocks of text, and I'm now building up a large data set of mo

I'm trying to learn Chinese. I wrote http://pingtype.github.io http://pingtype.github.io to parse blocks of text, and I'm now building up a large data set of movie subtitles, song lyrics, Bible translations, etc. Try reading this in TextWrangler: 1 . 教會組織: 小會:代議長老郭倍宏 The box causes the follow characters to be unreadable - it gets interpreted as a half-character. Deleting it makes the text show correctly. I tried it with ftfy, but it just copied the input through to the output.

rspeer 9y ago
Interesting. Here's the output of ftfy.explain_unicode on that text: U+0031 1 [Nd] DIGIT ONE U+0020 [Zs] SPACE U+002E . [Po] FULL STOP U+0020 [Zs] SPACE U+6559 教 [Lo] CJK UNIFIED IDEOGRAPH-6559 U+6703 會 [Lo] CJK UNIFIED IDEOGRAPH-6703 U+7D44 組 [Lo] CJK UNIFIED IDEOGRAPH-7D44 U+7E54 織 [Lo] CJK UNIFIED IDEOGRAPH-7E54 U+003A : [Po] COLON U+0020 [Zs] SPACE U+F081 \uf081 [Co] <unknown> U+5C0F 小 [Lo] CJK UNIFIED IDEOGRAPH-5C0F U+6703 會 [Lo] CJK UNIFIED IDEOGRAPH-6703 U+003A : [Po] COLON ... The anomalous character is U+F081, a character from the Private Use Area. TextWrangler is allowed to interpret it as whatever it wants, but I don't know why that would mess up all the following characters. Here's my theory. The text probably started out in the GBK encoding (used in mainland China). GBK has had different versions, which supported slightly different sets of characters. A number of these characters (decreasing as both GBK and Unicode updated) have no corresponding character in Unicode, and the standard thing to do when converting them to Unicode has been to convert them into Private Use characters. So that probably happened to this one, which may have started as a rare and inconsistently-supported character. Python's implementation of GBK (or GB18030) doesn't know what it is. So maybe what we need to do is flip through this Chinese technical standard [1], or maybe an older version of it, and track down which codepoint was historically mapped to U+F081 and what it is now and hahaha oh god [1] https://archive.org/stream/GB18030-2005/GB%2018030-2005#page/n0/mode/1up https://archive.org/stream/GB18030-2005/GB%2018030-2005#page...
jwong_ 9y ago
What character are you trying to use there? I can't see it on my browser either. Can you provide a screenshot of it?

rspeer 9y ago

Interesting. Here's the output of ftfy.explain_unicode on that text: U+0031 1 [Nd] DIGIT ONE U+0020 [Zs] SPACE U+002E . [Po] FULL STOP U+0020 [Zs] SPACE U+6559 教 [Lo] CJK UNIFIED IDEOGRAPH-6559 U+6703 會 [Lo] CJK UNIFIED IDEOGRAPH-6703 U+7D44 組 [Lo] CJK UNIFIED IDEOGRAPH-7D44 U+7E54 織 [Lo] CJK UNIFIED IDEOGRAPH-7E54 U+003A : [Po] COLON U+0020 [Zs] SPACE U+F081 \uf081 [Co] <unknown> U+5C0F 小 [Lo] CJK UNIFIED IDEOGRAPH-5C0F U+6703 會 [Lo] CJK UNIFIED IDEOGRAPH-6703 U+003A : [Po] COLON ... The anomalous character is U+F081, a character from the Private Use Area. TextWrangler is allowed to interpret it as whatever it wants, but I don't know why that would mess up all the following characters. Here's my theory. The text probably started out in the GBK encoding (used in mainland China). GBK has had different versions, which supported slightly different sets of characters. A number of these characters (decreasing as both GBK and Unicode updated) have no corresponding character in Unicode, and the standard thing to do when converting them to Unicode has been to convert them into Private Use characters. So that probably happened to this one, which may have started as a rare and inconsistently-supported character. Python's implementation of GBK (or GB18030) doesn't know what it is. So maybe what we need to do is flip through this Chinese technical standard [1], or maybe an older version of it, and track down which codepoint was historically mapped to U+F081 and what it is now and hahaha oh god [1] https://archive.org/stream/GB18030-2005/GB%2018030-2005#page/n0/mode/1up https://archive.org/stream/GB18030-2005/GB%2018030-2005#page...

jwong_ 9y ago

What character are you trying to use there? I can't see it on my browser either. Can you provide a screenshot of it?