3 ms·
It's a little strange for you to be criticizing ftfy as an encoding guesser, given that ftfy is not an encoding guesser. Are you thinking of chardet? > In an i
by rspeer 9y ago
It's a little strange for you to be criticizing ftfy as an encoding guesser, given that ftfy is not an encoding guesser. Are you thinking of chardet?
> In an ideal world, you would know what encoding bytes are in and could therefore decode them explicitly using the known correct encoding, and this library would be redundant.
Twitter is in a known encoding, UTF-8. Most of ftfy's examples come from Twitter. ftfy is not redundant.
When ftfy gets the input "#╨┐╤Ç╨╨╨╕╨╗╤î╨╜╨╛╨╡╨┐╨╕╤é╨╨╜╨╕╨╡", it's not because this tweet was somehow in a different encoding, it's because the bot that tweeted it literally tweeted "#╨┐╤Ç╨╨╨╕╨╗╤î╨╜╨╛╨╡╨┐╨╕╤é╨╨╜╨╕╨╡", in UTF-8, due to its own problems. So you decode the text that was tweeted from UTF-8, and then you start fixing it.
I still think you're thinking of chardet.
> If instead, as is often the case in the real world, the coding is unknown...
...then you will need to detect its encoding somehow. By now ftfy is a library for Python 3 only. If you try to pass bytes into the ftfy function, the Python language itself will stop you.
Are you hypothesizing that everyone dealing with unmarked bytes is passing them through a chain of chardet and ftfy, and blaming ftfy for all the problems that would result?
Incidentally, I do machine learning. (That's why I had to make ftfy, after all.) I have tried many machine learning solutions. They do not come close to ftfy's heuristics, which are designed to have extremely low false positive rates that are not attainable by ML. If you want one false positive per billion inputs... you're going to need like a quadrillion inputs, or you're going to need a lovingly hand-tuned heuristic.
- teddyh 9y ago> ftfy is not an encoding guesser If it isn’t an encoding guesser, what does it do that "".decode("encoding") doesn’t do?
- bulatb 9y agoA guesser answers the question: what encoding did they actually use? FTFY answers the question: What horrifying sequence of encode/decode transforms could output this sequence of bytes in UTF-8 that, when correctly decoded as UTF-8, still results in total gibberish? In other words... The problem fixed by an encoding guesser: 1. I encode my text with something that's not UTF-8-compatible. 2. I lie to you and say it's UTF-8. 3. You decode it as UTF-8 and get nonsense. What the heck? 4. A guesser tells you what encoding I actually used. 5. You decode it from the guessed encoding and get text. ---- The problem fixed by FTFY: 1. I encode string S with non-UTF-8 codec C. 2. I lie that it's UTF-8. 3. Someone decodes it as UTF-8. It's full of garbage, but they don't care. 4. They encode that sequence of nonsense symbols, not the original text, as UTF-8. Let's charitably name this "encoding" C'. 5. They say: Here teddyh, take this nice UTF-8. 6. You decode it as UTF-8. What the heck? 7. Is it ISO-8859? Some version of windows-X? Nope. It's UTF-8 carrying C', a non-encoding someone's broken algorithm made up on the spot. There's no decoder that can turn your UTF-8 back into the symbols of S, because the text you got was already garbage. 8. FTFY figures out what sequence of mismatched encode/decode steps generates text in C' and does the inverse, giving you back C^-1( C'^-1( C'( C( S )))) = S.