4 ms·

I think the article doesn't really clarify the reason these bugs exist. After all, is it really a bug? What if I so happen to be working with Chinese Unicode fi

by pierrec 11y ago

I think the article doesn't really clarify the reason these bugs exist. After all, is it really a bug? What if I so happen to be working with Chinese Unicode files without BOM that are also valid ASCII files? Then it would seem to me that Notepad was previously behaving correctly, and ever since Vista they introduced a bug! No, this isn't a low-level bug tied to any particular Windows API function. This is the inevitable result of the folly of trying to guess a file's encoding. When we find ourselves cornered into doing something this ridiculous, it becomes apparent that we, as a society of programmers, are extremely disorganized.

patio11 11y ago
What if I so happen to be working with Chinese Unicode files without BOM that are also valid ASCII files? What's the longest natural Chinese substring you can find for which this is the case? When we ran our encoding-and-language-detection heuristic developed for our research work I think, and this is 10+ years ago, it was 5 characters long -- there was no substring 6+ characters (12 bytes of UTF-16) which could be reasonably written in ASCII. This is because much of ASCII is unprintable, more-than-doubly so if you're concerned strictly with 7-bit ASCII as opposed to e.g. Latin-1. When's the last time you saw a string contain 0x18 ("cancel"), 0x07 ("bell"), etc? An assortment of 0x..18 characters for your amusement: 予,先,券,単,又. I picked ones which are included in commonly used words in Japanese. Seeing any one of these once in UTF-16 is dispositive that the bytestream is not ASCII. This problem is worth thinking about (which is why a customer asked a team of CS researchers and linguists to think about it, including one barely competent programmer who nonetheless had reasonably good intuitions for what character distributions looked like), but it turns out to be much, much less hard than many people originally expected. Anyhow, long story short: make a histogram of the bytes, dot product with the histogram signature you have of a few large corpora in decent-guess-based-on-apriori-knowledge languages/encoding, normalize. The screamingly obvious candidate is almost always the winner. If you want to do it really, really quickly you can even evaluate this heuristic with a Bloom filter in an FPGA.
- patio11 11y ago
  Elaboration since I like geeking out about this subject and demos like this always got the point across to non-Unicode-fluent developers. Here's a sample of Chinese language text, sourced by my favorite method: grab whatever is on Wikipedia's homepage. 2006年大西洋飓风季时间轴中记录有全年大西洋盆地所有热带和亚热带气旋形成、增强、减弱、登陆、转变成温带气旋以及消散的具体信息。2006年大西洋飓风季于2006年6月1日正式开始，同年11月30日结束，传统上这样的日期界定了一年中绝大多数热带气旋在大西洋形成的时间段，这一飓风季是继2001年大西洋飓风季以来第一个没有任何一场飓风在美国登陆的大西洋飓风季，也是继1994年大西洋飓风季以来第一次在整个十月份都没有热带气旋形成。美国国家飓风中心每年都会对前一年飓风季的所有天气系统进行重新分析，并根据结果更新其风暴数据库，因此时间轴中还包括实际操作中没有发布的信息。包括最大持续风速、位置、距离在内的所有数字都是经四舍五入换算成整数。2006年大西洋飓风季的活动程度与前一年相比远远不及。起初气象学家预计在极其活跃的2005年大西洋飓风季后，2006年的活动程度应该只会略低。然而，2006年迅速形成的厄尔尼诺-南方涛动现象、大西洋热带海域上空的撒哈拉空气层，以及以百慕大为中心的亚速尔高压这一强大二级高气压的持续存在，都令2006年大西洋飓风季的活动程度大幅降低。从10月2日以后一直到飓风季结束都完全没有热带气旋形成。2005年12月底形成的热带风暴泽塔一直持续到了2006年1月初，成为有纪录以来第二个跨日历年的大西洋风暴。虽然其存在时间不在任何一年飓风季的正式时间段里，但仍然可以视为2005和2006年大西洋飓风季的一部分。 Do you have an intuition for what that looks like if you interpret it as ASCII? No? Just guess: "almost plausibly an English document", "gibberish but mostly ASCII", "absolutely zero probability of being mistaken for ASCII." I whipped up a quick Ruby script: ``` require colorize; chinese = File.readlines("/tmp/chinese.txt"); puts chinese.bytes.map {|b| str = b.chr; if str.ascii_only? ? str.blue : str.red}.join ``` which converts that string from a Unicode encoding (UTF-8) to ASCII and renders the output blue where it collides with a printable ASCII character and as a red question mark otherwise. Did this match your prediction? https://www.evernote.com/l/Aaf93wCQGulAdZAtZjBA-8st_zgF_BKDlv8B/image.png https://www.evernote.com/l/Aaf93wCQGulAdZAtZjBA-8st_zgF_BKDl... If we first convert the string to UTF-16, it's a little less screamingly obvious but, well: https://www.evernote.com/l/AafWlxVe1CRIRKky5fDXLGYaVSFUetnXbmQB/image.png https://www.evernote.com/l/AafWlxVe1CRIRKky5fDXLGYaVSFUetnXb...
  - iSnow 11y ago
    Really love your posts, geeking it to the extremes about a strange and rare problem. thanks :)
  - StavrosK 11y ago
    Your post confuses me. The first screenshot is pretty much exactly how I would translate the Chinese string as well.
- rspeer 11y ago
pif 11y ago
> This is the inevitable result of the folly of trying to guess a file's encoding. I'd say just: "This is the inevitable result of the folly of trying to guess." Guessing without confirmation request has no place in system libraries, nor in any "serious" (read: money-related) application. Guessing is OK for providing nice defaults that work most of the time, but the user should have the final word.

by pierrec 11y ago

patio11 11y ago

What if I so happen to be working with Chinese Unicode files without BOM that are also valid ASCII files? What's the longest natural Chinese substring you can find for which this is the case? When we ran our encoding-and-language-detection heuristic developed for our research work I think, and this is 10+ years ago, it was 5 characters long -- there was no substring 6+ characters (12 bytes of UTF-16) which could be reasonably written in ASCII. This is because much of ASCII is unprintable, more-than-doubly so if you're concerned strictly with 7-bit ASCII as opposed to e.g. Latin-1. When's the last time you saw a string contain 0x18 ("cancel"), 0x07 ("bell"), etc? An assortment of 0x..18 characters for your amusement: 予,先,券,単,又. I picked ones which are included in commonly used words in Japanese. Seeing any one of these once in UTF-16 is dispositive that the bytestream is not ASCII. This problem is worth thinking about (which is why a customer asked a team of CS researchers and linguists to think about it, including one barely competent programmer who nonetheless had reasonably good intuitions for what character distributions looked like), but it turns out to be much, much less hard than many people originally expected. Anyhow, long story short: make a histogram of the bytes, dot product with the histogram signature you have of a few large corpora in decent-guess-based-on-apriori-knowledge languages/encoding, normalize. The screamingly obvious candidate is almost always the winner. If you want to do it really, really quickly you can even evaluate this heuristic with a Bloom filter in an FPGA.

patio11 11y ago
Elaboration since I like geeking out about this subject and demos like this always got the point across to non-Unicode-fluent developers. Here's a sample of Chinese language text, sourced by my favorite method: grab whatever is on Wikipedia's homepage. 2006年大西洋飓风季时间轴中记录有全年大西洋盆地所有热带和亚热带气旋形成、增强、减弱、登陆、转变成温带气旋以及消散的具体信息。2006年大西洋飓风季于2006年6月1日正式开始，同年11月30日结束，传统上这样的日期界定了一年中绝大多数热带气旋在大西洋形成的时间段，这一飓风季是继2001年大西洋飓风季以来第一个没有任何一场飓风在美国登陆的大西洋飓风季，也是继1994年大西洋飓风季以来第一次在整个十月份都没有热带气旋形成。美国国家飓风中心每年都会对前一年飓风季的所有天气系统进行重新分析，并根据结果更新其风暴数据库，因此时间轴中还包括实际操作中没有发布的信息。包括最大持续风速、位置、距离在内的所有数字都是经四舍五入换算成整数。2006年大西洋飓风季的活动程度与前一年相比远远不及。起初气象学家预计在极其活跃的2005年大西洋飓风季后，2006年的活动程度应该只会略低。然而，2006年迅速形成的厄尔尼诺-南方涛动现象、大西洋热带海域上空的撒哈拉空气层，以及以百慕大为中心的亚速尔高压这一强大二级高气压的持续存在，都令2006年大西洋飓风季的活动程度大幅降低。从10月2日以后一直到飓风季结束都完全没有热带气旋形成。2005年12月底形成的热带风暴泽塔一直持续到了2006年1月初，成为有纪录以来第二个跨日历年的大西洋风暴。虽然其存在时间不在任何一年飓风季的正式时间段里，但仍然可以视为2005和2006年大西洋飓风季的一部分。 Do you have an intuition for what that looks like if you interpret it as ASCII? No? Just guess: "almost plausibly an English document", "gibberish but mostly ASCII", "absolutely zero probability of being mistaken for ASCII." I whipped up a quick Ruby script: ``` require colorize; chinese = File.readlines("/tmp/chinese.txt"); puts chinese.bytes.map {|b| str = b.chr; if str.ascii_only? ? str.blue : str.red}.join ``` which converts that string from a Unicode encoding (UTF-8) to ASCII and renders the output blue where it collides with a printable ASCII character and as a red question mark otherwise. Did this match your prediction? https://www.evernote.com/l/Aaf93wCQGulAdZAtZjBA-8st_zgF_BKDlv8B/image.png https://www.evernote.com/l/Aaf93wCQGulAdZAtZjBA-8st_zgF_BKDl... If we first convert the string to UTF-16, it's a little less screamingly obvious but, well: https://www.evernote.com/l/AafWlxVe1CRIRKky5fDXLGYaVSFUetnXbmQB/image.png https://www.evernote.com/l/AafWlxVe1CRIRKky5fDXLGYaVSFUetnXb...
- iSnow 11y ago
  Really love your posts, geeking it to the extremes about a strange and rare problem. thanks :)
- StavrosK 11y ago
  Your post confuses me. The first screenshot is pretty much exactly how I would translate the Chinese string as well.
rspeer 11y ago

pif 11y ago

> This is the inevitable result of the folly of trying to guess a file's encoding. I'd say just: "This is the inevitable result of the folly of trying to guess." Guessing without confirmation request has no place in system libraries, nor in any "serious" (read: money-related) application. Guessing is OK for providing nice defaults that work most of the time, but the user should have the final word.