3 ms·

I haven't seen discussion of this point yet, but the post completely fails to provide any data to back up its assertion that charset detection heuristics works,

I haven't seen discussion of this point yet, but the post completely fails to provide any data to back up its assertion that charset detection heuristics works, because the feedback I've seen from people who actually work with charsets is that it largely doesn't (especially if you're based on naive one-byte frequency analysis). Okay, sure, it works if you want to distinguish between KOI8-R and Windows-1252, but what about Windows-1252 and Windows-1257? See for example this effort in building a universal charset detector in Gecko: https://bugzilla.mozilla.org/show_bug.cgi?id=1551276 https://bugzilla.mozilla.org/show_bug.cgi?id=1551276

toast0 3y ago
I've done some charset detection, although it's been a while. Heuristics kind of work for somethings --- I'm a big fan of if it's decodable as utf-8, it's probably utf-8, unless there's zero bytes (in most text). If there's a lot of zero bytes, maybe it's UCS-2 or UTF-16, and you can try to figure out the byte order and if it decodes as utf-16. If it doesn't fit in those categories, you've got a much harder guessing game. But usually you can't actually ask the source what it is, because they probably don't know and might not understand the question or might not be contactable. Usually, you have to guess something, so you may as well take someone else's work to guess, if you don't have better information.
renhanxue 3y ago
Yeah. The fantastic python library ftfy ("fixes text for you", https://ftfy.readthedocs.io/en/latest/index.html https://ftfy.readthedocs.io/en/latest/index.html), designed to fix mangled Unicode (mojibake, of many different varieties), mentions in its docs that heuristic encoding guessers are the cause of many of the problems ftfy is designed to fix. It's magical, by the way. It has an entire section of its documentation dedicated to why you should not use heuristic charset detection: https://ftfy.readthedocs.io/en/latest/avoid.html#don-t-use-chardet https://ftfy.readthedocs.io/en/latest/avoid.html#don-t-use-c...
- torstenvl 3y ago
  That section explains why not to use a specific naive charset detection library that doesn't have a strong prior for UTF-8. There's no basis for extrapolating that further.

toast0 3y ago

I've done some charset detection, although it's been a while. Heuristics kind of work for somethings --- I'm a big fan of if it's decodable as utf-8, it's probably utf-8, unless there's zero bytes (in most text). If there's a lot of zero bytes, maybe it's UCS-2 or UTF-16, and you can try to figure out the byte order and if it decodes as utf-16. If it doesn't fit in those categories, you've got a much harder guessing game. But usually you can't actually ask the source what it is, because they probably don't know and might not understand the question or might not be contactable. Usually, you have to guess something, so you may as well take someone else's work to guess, if you don't have better information.

renhanxue 3y ago

Yeah. The fantastic python library ftfy ("fixes text for you", https://ftfy.readthedocs.io/en/latest/index.html https://ftfy.readthedocs.io/en/latest/index.html), designed to fix mangled Unicode (mojibake, of many different varieties), mentions in its docs that heuristic encoding guessers are the cause of many of the problems ftfy is designed to fix. It's magical, by the way. It has an entire section of its documentation dedicated to why you should not use heuristic charset detection: https://ftfy.readthedocs.io/en/latest/avoid.html#don-t-use-chardet https://ftfy.readthedocs.io/en/latest/avoid.html#don-t-use-c...

torstenvl 3y ago
That section explains why not to use a specific naive charset detection library that doesn't have a strong prior for UTF-8. There's no basis for extrapolating that further.