3 ms·

I couldn't accuse you of hating fun, but I could accuse you of hating thorough documentation. Would you prefer I hadn't told you about the one case in six milli

by rspeer 12y ago

I couldn't accuse you of hating fun, but I could accuse you of hating thorough documentation. Would you prefer I hadn't told you about the one case in six million where the library fails? And that failure is really an amazing coincidence when you look into it; change any one of those line-drawing characters and it'll be fine. Keep in mind that this is a library that finds emoji that has been damaged by "serious" software written by, let's call them "Basic Multilingual Plane neanderthals", and puts it back. There's your fun.

teddyh 12y ago
You brought up the fact that the library was only for (your word) “serious” text. You chose the specific example of “non-serious” text. You can hardly fault me, then, for accusing the library of being incompatible with non-seriousness.
- unicodedammit 12y ago
  You're being unnecessarily argumentative. The web is filled with badly-encoded/re-encoded/mixed-encoded text. One of the worst offenders is Microsoft Outlook, which sends emails using local 8-bit windows codepage by default, instead of utf-8. Pass that through several mail gateways, display those messages on some php forum, and you get yourself a bad mess that cannot be decoded successfully by any python codec. chardet is useless in that case - there is no 'valid' encoding per se. This library is taking the only possible approach, which is to segment the text and try to convert each segment into its most probable unicode representation. It seems to cover a larger number of encoding mixups compared to other libraries, that's great! Comment to rspeer: in fix_text_segment(), I would limit the number of recursive passes on the text to 5-10. Right now it's using 'while True', which might take a very long time to converge on corrupted/binary data.
  - rspeer 12y ago
    So, actually, it seemed to make sense at first to limit it to, say, 2 or 3 passes. But then I read about Spotify's username exploit [1]. That made it pretty clear to me that any Unicode-transforming function should be idempotent whenever possible, so that you never end up with inconsistent answers about whether strings are equivalent. I have also seen text that was encoded six times in UTF-8 (and decoded five times in Windows-1252). Although ftfy had to leave it as is; it didn't successfully decode because it was truncated. [1] http://labs.spotify.com/2013/06/18/creative-usernames/ http://labs.spotify.com/2013/06/18/creative-usernames/
  - teddyh 12y ago
    > This library is taking the only possible approach It is taking the only possible approach if we assume that it must use one and only one algorithm for all uses. Otherwise, it seems to me that a lot of careful tuning and configuration would be needed in order for this library to make the best guesses it possibly can make for a specific user’s situation and data. > might take a very long time to converge A limit there might be appropriate – otherwise there might exist a “billion laughs” style attack.

by rspeer 12y ago

teddyh 12y ago

You brought up the fact that the library was only for (your word) “serious” text. You chose the specific example of “non-serious” text. You can hardly fault me, then, for accusing the library of being incompatible with non-seriousness.

unicodedammit 12y ago
You're being unnecessarily argumentative. The web is filled with badly-encoded/re-encoded/mixed-encoded text. One of the worst offenders is Microsoft Outlook, which sends emails using local 8-bit windows codepage by default, instead of utf-8. Pass that through several mail gateways, display those messages on some php forum, and you get yourself a bad mess that cannot be decoded successfully by any python codec. chardet is useless in that case - there is no 'valid' encoding per se. This library is taking the only possible approach, which is to segment the text and try to convert each segment into its most probable unicode representation. It seems to cover a larger number of encoding mixups compared to other libraries, that's great! Comment to rspeer: in fix_text_segment(), I would limit the number of recursive passes on the text to 5-10. Right now it's using 'while True', which might take a very long time to converge on corrupted/binary data.
- rspeer 12y ago
  So, actually, it seemed to make sense at first to limit it to, say, 2 or 3 passes. But then I read about Spotify's username exploit [1]. That made it pretty clear to me that any Unicode-transforming function should be idempotent whenever possible, so that you never end up with inconsistent answers about whether strings are equivalent. I have also seen text that was encoded six times in UTF-8 (and decoded five times in Windows-1252). Although ftfy had to leave it as is; it didn't successfully decode because it was truncated. [1] http://labs.spotify.com/2013/06/18/creative-usernames/ http://labs.spotify.com/2013/06/18/creative-usernames/
- teddyh 12y ago
  > This library is taking the only possible approach It is taking the only possible approach if we assume that it must use one and only one algorithm for all uses. Otherwise, it seems to me that a lot of careful tuning and configuration would be needed in order for this library to make the best guesses it possibly can make for a specific user’s situation and data. > might take a very long time to converge A limit there might be appropriate – otherwise there might exist a “billion laughs” style attack.