3 ms·

I think you're criticizing the wrong library. ftfy isn't about encoding detection. Should I take guess_bytes out of the documentation to stop giving that impres

by rspeer 12y ago

I think you're criticizing the wrong library. ftfy isn't about encoding detection. Should I take guess_bytes out of the documentation to stop giving that impression? It's the library you use when the data you get has already been decoded incorrectly. The user of ftfy cares about encodings, but gets data from sources that don't. And in no practical sense does it corrupt your data. I don't know where you got that idea from. It leaves good data alone. I will not say that false positives are nonexistent, but they are vanishingly rare -- see http://ftfy.readthedocs.org/en/latest/#accuracy http://ftfy.readthedocs.org/en/latest/#accuracy -- and they don't occur in "serious" data, they occur when people are screwing around with bizarre emoticons and stuff.

teddyh 12y ago
If a string like "&" is supplied to ftfy, it will probably guess or detect that this is HTML encoded. Therefore, ftfy detects encodings. If the guess is wrong, the data has now been corrupted. Your protestations about how corruptions will not occur in “serious” data (what is is that, anyway?), and blaming “bizarre emoticons” is exactly symptomatic of what I’m talking about – you are blaming every wrong guess which this library makes on uses of non-ASCII. This is being an ASCII neanderthal. The problems of ignoring encodings are real, and should not be blamed on users of “bizarre emoticons and stuff”.
- rspeer 12y ago
  fix_entities is a parameter. It's not guessing, you told it. You could vaguely criticize the fix_entities='auto' setting as a guess, except it's a guess that's only wrong if you manage to provide it an HTML document with zero tags in it. An example of a false positive is "├┤a┼┐a┼┐a┼┐a┼┐a". That is what I mean by non-serious text. False positives will always exist, and you should appreciate that I'm testing on millions of examples to find out what they are. Your suggested machine learning approach would never get to 99.999984% precision. Using non-ASCII is totally fine, and this library would have no purpose in an all-ASCII world.
  - teddyh 12y ago
    > you told it. Another commenter claims that this worked: > ftfy.fix_text('L&amp;Atilde;&amp;sup3;pez') There is no “fix_entities = True” there. Indeed, if the library would require such a parameter, what would be the point? If you already know the encoding, the library has no reason to exist. Therefore, the whole point of the library is to guess the encoding. > you should appreciate that I'm testing on millions of examples Examples taken, I would assume, either from your personal use cases, the use cases of your customers, or some sort of general grab-bag of mis-encoded text you could find. I would assume that this one-size-fits-all ad-hoc rule set would be wrong for many users in their specialized use cases, and would bite them when they least expect it.
    - rspeer 12y ago
      Examples are taken from Twitter's live stream. You're not even reading the documentation, you're just searching for reasons to call me an "ASCII neanderthal" over a library that an ASCII neanderthal would have no use for. And I fail to see how the default settings being able to fix 'L&amp;Atilde;&amp;sup3;pez' is anything but a resounding success.
      - teddyh 12y ago
        Would that not bias your rules towards decoding only the errors which are made by all existing Twitter clients (of which I understand there are relatively few)? Anyway, now we’re just arguing in circles. I said that the library would guess that “&” was HTML encoded. You said “fix_entities is a parameter. It's not guessing, you told it.” I said that the example given has no such parameter. You then turned around and said that it was a success that it guessed correctly, but my point was that it was, indeed, guessing, and might, therefore, guess wrong. I don’t want to call you an ASCII neanderthal, really, and I’m sorry I did, it’s just that your library helps the actual ASCII neanderthals from having to bother with evolving. This is my main complaint about this library. It will probably be used by them, reflexively, to decode everything, even when the encoding is known, and therefore introduce (admittedly relatively small amounts of) data corruption (but these things have a tendency to crop up when you least wanted them). Whereas if your library was not used, users would have to think about what decoding to use, use it, and not introduce data corruption. Also, I have some misgivings about a one-size-fits-all solution of guessing encodings – I guess that it would never really work in quite the painless way most your users imagine. To solve this, I advocated a user-customizable training approach, which would, for each user, be the best possible one for their use case. It would also have the beneficial side effect of forcing the users to actually think about their data and what encodings it was likely to have and in what circumstances, thus making them evolve to Homo Unicodus. ☺ Of course, I could be wrong about this, but my principal worry about this library, as stated above, remains.
teddyh 12y ago
> It's the library you use when the data you get has already been decoded incorrectly. Wait, does this not make the usability of this library very limited? How would I know that something has been decoded incorrectly? If I’m already handling this manually, what is the point of having a library?
- rspeer 12y ago
  Having lost the plot of your other argument, you fall back on the time-honored low-effort HN comment, "Isn't this useless?" Your questions could be answered, but not by me. Plonk.
  - belorn 12y ago
    Vuze, a very commonly used torrent client has to take encoded torrent titles and decode them. Sometimes the decoding fails, because of corruption or missing encoding type. Their approach is then to attempt each type of encoding they got, ignoring errors, and show the result to the user so the user can decide (detect) which decoding worked and which didn't. Can ftfy replace this functionality? Is it doing the very encoding detection which is currently done by humans?
    - rspeer 12y ago
      That sounds like a use case where the manual intervention is pretty important. That's encoding detection, where sometimes you have unmarked bytes. I would not, in fact, recommend ftfy there, and the README warns you against using it in that case. There's some auto-detection you can do -- for example, you can distinguish UTF-8 from byte-order-marked UTF-16 with 100% accuracy, by design. You could also try chardet if you're okay with some amount of errors. Maybe show the detected encoding first. Cases where ftfy is useful: * Web scraping -- sometimes you get data that decodes in the encoding it claims to be in, but isn't quite right * Handling data that has, at one point, been imported and exported in Microsoft Office, without every user consistently picking exactly the right format from like 20 inaccurately-named options * Handling data that was stored by half-assed services written in, say, PHP, and not tested outside of ASCII * Reading CESU-8 (the non-standard encoding that Java and MySQL call "UTF-8", for backward compatibility) in Python without breaking it even more. (This isn't automatic.) * Handling data that's combined from multiple sources, in mixed encodings * All the other situations in which mojibake arises in mostly-readable text, which there seem to be no end of.
fnl 12y ago
Well, teddyh might have a point here, nonetheless: By now, I understand that ftfy is about fixing mixed up encodings between UTF8, latin-1, CP437, CP125[12] and MacRoman (only). But by claiming you are fixing "Unicode" in general as the first thing on the GitHub page, you might be misleading first-time visitors. Maybe you should try to place the "warning" about the encodings your library does handle right at the start somewhere? And make it clear that "moji-un-baking" is the library's central and main use-case, not just an "interesting thing" it can do. Despite being quite aware of Unicode and string encoding, I had exactly the same thoughts as teddyh as I read the first few paragraphs ("Oh, now we will see those encoding illiterates converting all those beautiful bytes in some highly informative character encoding to all-too-boring-ASCII.") Which leads me to my other concern: Why do you use NFKC compatibility as the default normalization? Given you are a text mining company, you of all guys should know you loose valuable information - particularly about numbers, super- and subscript characters - with this normalization strategy. Doing NFKC on stuff like all kinds of articles, books, patents, etc. would lead to potentially disastrous results (e.g., NFKC "decomposes" the string 'O\u2082\u00B9' to 'O21' instead of 'O_2^1' - "oxygen, reference 1"). In general, I think NFC is what Python and many other libraries do, while I believe NFKC should only be used when you know what you are doing (and why you need it). Maybe it is useful for some strange, geeky tweets, but I would argue that its the corner case, not the default.
- rspeer 12y ago
  I wonder if I could change the default to NFC in the next version without breaking people's expectations. It is a safer default. When it comes to text analytics, the underlying tagger and stuff won't know what O21 is any more than it knows what O_2^1 is anyway. And NFKC is useful for mixed Latin and Japanese text, which I wouldn't entirely dismiss as strange and geeky. But it's true that the default could be more conservative.

by rspeer 12y ago

teddyh 12y ago

If a string like "&" is supplied to ftfy, it will probably guess or detect that this is HTML encoded. Therefore, ftfy detects encodings. If the guess is wrong, the data has now been corrupted. Your protestations about how corruptions will not occur in “serious” data (what is is that, anyway?), and blaming “bizarre emoticons” is exactly symptomatic of what I’m talking about – you are blaming every wrong guess which this library makes on uses of non-ASCII. This is being an ASCII neanderthal. The problems of ignoring encodings are real, and should not be blamed on users of “bizarre emoticons and stuff”.

rspeer 12y ago
fix_entities is a parameter. It's not guessing, you told it. You could vaguely criticize the fix_entities='auto' setting as a guess, except it's a guess that's only wrong if you manage to provide it an HTML document with zero tags in it. An example of a false positive is "├┤a┼┐a┼┐a┼┐a┼┐a". That is what I mean by non-serious text. False positives will always exist, and you should appreciate that I'm testing on millions of examples to find out what they are. Your suggested machine learning approach would never get to 99.999984% precision. Using non-ASCII is totally fine, and this library would have no purpose in an all-ASCII world.
- teddyh 12y ago
  > you told it. Another commenter claims that this worked: > ftfy.fix_text('L&amp;Atilde;&amp;sup3;pez') There is no “fix_entities = True” there. Indeed, if the library would require such a parameter, what would be the point? If you already know the encoding, the library has no reason to exist. Therefore, the whole point of the library is to guess the encoding. > you should appreciate that I'm testing on millions of examples Examples taken, I would assume, either from your personal use cases, the use cases of your customers, or some sort of general grab-bag of mis-encoded text you could find. I would assume that this one-size-fits-all ad-hoc rule set would be wrong for many users in their specialized use cases, and would bite them when they least expect it.
  - rspeer 12y ago
    Examples are taken from Twitter's live stream. You're not even reading the documentation, you're just searching for reasons to call me an "ASCII neanderthal" over a library that an ASCII neanderthal would have no use for. And I fail to see how the default settings being able to fix 'L&amp;Atilde;&amp;sup3;pez' is anything but a resounding success.
    - teddyh 12y ago
      Would that not bias your rules towards decoding only the errors which are made by all existing Twitter clients (of which I understand there are relatively few)? Anyway, now we’re just arguing in circles. I said that the library would guess that “&” was HTML encoded. You said “fix_entities is a parameter. It's not guessing, you told it.” I said that the example given has no such parameter. You then turned around and said that it was a success that it guessed correctly, but my point was that it was, indeed, guessing, and might, therefore, guess wrong. I don’t want to call you an ASCII neanderthal, really, and I’m sorry I did, it’s just that your library helps the actual ASCII neanderthals from having to bother with evolving. This is my main complaint about this library. It will probably be used by them, reflexively, to decode everything, even when the encoding is known, and therefore introduce (admittedly relatively small amounts of) data corruption (but these things have a tendency to crop up when you least wanted them). Whereas if your library was not used, users would have to think about what decoding to use, use it, and not introduce data corruption. Also, I have some misgivings about a one-size-fits-all solution of guessing encodings – I guess that it would never really work in quite the painless way most your users imagine. To solve this, I advocated a user-customizable training approach, which would, for each user, be the best possible one for their use case. It would also have the beneficial side effect of forcing the users to actually think about their data and what encodings it was likely to have and in what circumstances, thus making them evolve to Homo Unicodus. ☺ Of course, I could be wrong about this, but my principal worry about this library, as stated above, remains.

teddyh 12y ago

> It's the library you use when the data you get has already been decoded incorrectly. Wait, does this not make the usability of this library very limited? How would I know that something has been decoded incorrectly? If I’m already handling this manually, what is the point of having a library?

rspeer 12y ago
Having lost the plot of your other argument, you fall back on the time-honored low-effort HN comment, "Isn't this useless?" Your questions could be answered, but not by me. Plonk.
- belorn 12y ago
  Vuze, a very commonly used torrent client has to take encoded torrent titles and decode them. Sometimes the decoding fails, because of corruption or missing encoding type. Their approach is then to attempt each type of encoding they got, ignoring errors, and show the result to the user so the user can decide (detect) which decoding worked and which didn't. Can ftfy replace this functionality? Is it doing the very encoding detection which is currently done by humans?
  - rspeer 12y ago
    That sounds like a use case where the manual intervention is pretty important. That's encoding detection, where sometimes you have unmarked bytes. I would not, in fact, recommend ftfy there, and the README warns you against using it in that case. There's some auto-detection you can do -- for example, you can distinguish UTF-8 from byte-order-marked UTF-16 with 100% accuracy, by design. You could also try chardet if you're okay with some amount of errors. Maybe show the detected encoding first. Cases where ftfy is useful: * Web scraping -- sometimes you get data that decodes in the encoding it claims to be in, but isn't quite right * Handling data that has, at one point, been imported and exported in Microsoft Office, without every user consistently picking exactly the right format from like 20 inaccurately-named options * Handling data that was stored by half-assed services written in, say, PHP, and not tested outside of ASCII * Reading CESU-8 (the non-standard encoding that Java and MySQL call "UTF-8", for backward compatibility) in Python without breaking it even more. (This isn't automatic.) * Handling data that's combined from multiple sources, in mixed encodings * All the other situations in which mojibake arises in mostly-readable text, which there seem to be no end of.

fnl 12y ago

Well, teddyh might have a point here, nonetheless: By now, I understand that ftfy is about fixing mixed up encodings between UTF8, latin-1, CP437, CP125[12] and MacRoman (only). But by claiming you are fixing "Unicode" in general as the first thing on the GitHub page, you might be misleading first-time visitors. Maybe you should try to place the "warning" about the encodings your library does handle right at the start somewhere? And make it clear that "moji-un-baking" is the library's central and main use-case, not just an "interesting thing" it can do. Despite being quite aware of Unicode and string encoding, I had exactly the same thoughts as teddyh as I read the first few paragraphs ("Oh, now we will see those encoding illiterates converting all those beautiful bytes in some highly informative character encoding to all-too-boring-ASCII.") Which leads me to my other concern: Why do you use NFKC compatibility as the default normalization? Given you are a text mining company, you of all guys should know you loose valuable information - particularly about numbers, super- and subscript characters - with this normalization strategy. Doing NFKC on stuff like all kinds of articles, books, patents, etc. would lead to potentially disastrous results (e.g., NFKC "decomposes" the string 'O\u2082\u00B9' to 'O21' instead of 'O_2^1' - "oxygen, reference 1"). In general, I think NFC is what Python and many other libraries do, while I believe NFKC should only be used when you know what you are doing (and why you need it). Maybe it is useful for some strange, geeky tweets, but I would argue that its the corner case, not the default.

rspeer 12y ago
I wonder if I could change the default to NFC in the next version without breaking people's expectations. It is a safer default. When it comes to text analytics, the underlying tagger and stuff won't know what O21 is any more than it knows what O_2^1 is anyway. And NFKC is useful for mixed Latin and Japanese text, which I wouldn't entirely dismiss as strange and geeky. But it's true that the default could be more conservative.