2 ms·
Vuze, a very commonly used torrent client has to take encoded torrent titles and decode them. Sometimes the decoding fails, because of corruption or missing enc
by belorn 12y ago
Vuze, a very commonly used torrent client has to take encoded torrent titles and decode them. Sometimes the decoding fails, because of corruption or missing encoding type.
Their approach is then to attempt each type of encoding they got, ignoring errors, and show the result to the user so the user can decide (detect) which decoding worked and which didn't.
Can ftfy replace this functionality? Is it doing the very encoding detection which is currently done by humans?
- rspeer 12y agoThat sounds like a use case where the manual intervention is pretty important. That's encoding detection, where sometimes you have unmarked bytes. I would not, in fact, recommend ftfy there, and the README warns you against using it in that case. There's some auto-detection you can do -- for example, you can distinguish UTF-8 from byte-order-marked UTF-16 with 100% accuracy, by design. You could also try chardet if you're okay with some amount of errors. Maybe show the detected encoding first. Cases where ftfy is useful: * Web scraping -- sometimes you get data that decodes in the encoding it claims to be in, but isn't quite right * Handling data that has, at one point, been imported and exported in Microsoft Office, without every user consistently picking exactly the right format from like 20 inaccurately-named options * Handling data that was stored by half-assed services written in, say, PHP, and not tested outside of ASCII * Reading CESU-8 (the non-standard encoding that Java and MySQL call "UTF-8", for backward compatibility) in Python without breaking it even more. (This isn't automatic.) * Handling data that's combined from multiple sources, in mixed encodings * All the other situations in which mojibake arises in mostly-readable text, which there seem to be no end of.