Interesting. Here's the output of ftfy.explain_unicode on that text: U+0031 1 [Nd] DIGIT ONE U+0020 [Zs] SPACE U+002E . [Po]

Interesting. Here's the output of ftfy.explain_unicode on that text: U+0031 1 [Nd] DIGIT ONE U+0020 [Zs] SPACE U+002E . [Po] FULL STOP U+0020 [Zs] SPACE U+6559 教 [Lo] CJK UNIFIED IDEOGRAPH-6559 U+6703 會 [Lo] CJK UNIFIED IDEOGRAPH-6703 U+7D44 組 [Lo] CJK UNIFIED IDEOGRAPH-7D44 U+7E54 織 [Lo] CJK UNIFIED IDEOGRAPH-7E54 U+003A : [Po] COLON U+0020 [Zs] SPACE U+F081 \uf081 [Co] <unknown> U+5C0F 小 [Lo] CJK UNIFIED IDEOGRAPH-5C0F U+6703 會 [Lo] CJK UNIFIED IDEOGRAPH-6703 U+003A : [Po] COLON ... The anomalous character is U+F081, a character from the Private Use Area. TextWrangler is allowed to interpret it as whatever it wants, but I don't know why that would mess up all the following characters. Here's my theory. The text probably started out in the GBK encoding (used in mainland China). GBK has had different versions, which supported slightly different sets of characters. A number of these characters (decreasing as both GBK and Unicode updated) have no corresponding character in Unicode, and the standard thing to do when converting them to Unicode has been to convert them into Private Use characters. So that probably happened to this one, which may have started as a rare and inconsistently-supported character. Python's implementation of GBK (or GB18030) doesn't know what it is. So maybe what we need to do is flip through this Chinese technical standard [1], or maybe an older version of it, and track down which codepoint was historically mapped to U+F081 and what it is now and hahaha oh god [1] https://archive.org/stream/GB18030-2005/GB%2018030-2005#page/n0/mode/1up https://archive.org/stream/GB18030-2005/GB%2018030-2005#page...