5 ms·
I can’t help but think that this merely gives people the excuse they need for not understanding this “Things-that-are-not-ASCII” problem. Using this library is
by teddyh 12y ago
I can’t help but think that this merely gives people the excuse they need for not understanding this “Things-that-are-not-ASCII” problem. Using this library is a desperate attempt to have a just-fix-it function, but it can never cover all cases, and will inevitably corrupt data. To use this library is to remain an ASCII neanderthal, ignorant of the modern world and the difference of text, bytes and encodings.
Let me explain in some detail why this library is not a good thing:
In an ideal world, you would know what encoding bytes are in and could therefore decode them explicitly using the known correct encoding, and this library would be redundant.
If instead, as is often the case in the real world, the coding is unknown, there exists the question of how to resolve the numerous ambiguities which result. A library such as this would have to guess what encoding to use in each specific instance, and the choices it ideally should make are extremely dependent on the circumstances and even the immediate context. As it is, the library is hard-coded with some specific algorithms to choose some encodings over others, and if those assumptions does not match your use case exactly, the library will corrupt your data.
A much better solution would perhaps involve a machine learning solution to the problem, and having the library be trained to deduce the probable encodings from a large set of example data from each user’s individual use case. Even these will occasionally be wrong, but at least it would be the best we could do with unknown encodings without resorting to manual processing.
However, a one-size-fits-all “solution” such as this is merely giving people a further excuse to keep not caring about encodings, to pretend that encodings can be “detected”, and that there exists such a thing as “plain text”.
- rspeer 12y agoI think you're criticizing the wrong library. ftfy isn't about encoding detection. Should I take guess_bytes out of the documentation to stop giving that impression? It's the library you use when the data you get has already been decoded incorrectly. The user of ftfy cares about encodings, but gets data from sources that don't. And in no practical sense does it corrupt your data. I don't know where you got that idea from. It leaves good data alone. I will not say that false positives are nonexistent, but they are vanishingly rare -- see http://ftfy.readthedocs.org/en/latest/#accuracy http://ftfy.readthedocs.org/en/latest/#accuracy -- and they don't occur in "serious" data, they occur when people are screwing around with bizarre emoticons and stuff.
- teddyh 12y agoIf a string like "&" is supplied to ftfy, it will probably guess or detect that this is HTML encoded. Therefore, ftfy detects encodings. If the guess is wrong, the data has now been corrupted. Your protestations about how corruptions will not occur in “serious” data (what is is that, anyway?), and blaming “bizarre emoticons” is exactly symptomatic of what I’m talking about – you are blaming every wrong guess which this library makes on uses of non-ASCII. This is being an ASCII neanderthal. The problems of ignoring encodings are real, and should not be blamed on users of “bizarre emoticons and stuff”.
- rspeer 12y agofix_entities is a parameter. It's not guessing, you told it. You could vaguely criticize the fix_entities='auto' setting as a guess, except it's a guess that's only wrong if you manage to provide it an HTML document with zero tags in it. An example of a false positive is "├┤a┼┐a┼┐a┼┐a┼┐a". That is what I mean by non-serious text. False positives will always exist, and you should appreciate that I'm testing on millions of examples to find out what they are. Your suggested machine learning approach would never get to 99.999984% precision. Using non-ASCII is totally fine, and this library would have no purpose in an all-ASCII world.
- teddyh 12y ago> you told it. Another commenter claims that this worked: > ftfy.fix_text('López') There is no “fix_entities = True” there. Indeed, if the library would require such a parameter, what would be the point? If you already know the encoding, the library has no reason to exist. Therefore, the whole point of the library is to guess the encoding. > you should appreciate that I'm testing on millions of examples Examples taken, I would assume, either from your personal use cases, the use cases of your customers, or some sort of general grab-bag of mis-encoded text you could find. I would assume that this one-size-fits-all ad-hoc rule set would be wrong for many users in their specialized use cases, and would bite them when they least expect it.
- 12y ago