6 ms·

Python-ftfy: Given Unicode text, make its representation possibly less broken

101 points by rspeer 12y ago

tdumitrescu 12y ago
Wow, recruiting pitch at the top of the README. Someone make Adblock for Github.
- artursapek 12y ago
  I think the least we can give in return to companies that open-source useful stuff like this is look the other way when they self-promote a little bit.
- jjgreen 12y ago
  I clicked on the careers link and would seriously consider working for a company with such an enlightened attitude to open source, chill! (In fact in the wrong country, but nevertheless...)
meowface 12y ago
This is a great project because a lot of work has been put into finding solutions to many different edge cases, and it "just works".
rwg 12y ago
It even successfully un-mangles the shipping label from Ode to a Shipping Label: https://www.facebook.com/cmb/posts/619241744770551:0 https://www.facebook.com/cmb/posts/619241744770551:0 >>> ftfy.fix_text('L&amp;Atilde;&amp;sup3;pez') 'López' Bravo!
unicodedammit 12y ago
Very common problem in web scraping, for example forum site might contain a mix of MacRoman, Windows codepage and various european codepages in a single page (yes, even in 2014!). Seems like a more advanced version of UnicodeDammit module ( http://www.crummy.com/software/BeautifulSoup/bs4/doc/#unicode-dammit http://www.crummy.com/software/BeautifulSoup/bs4/doc/#unicod... ). Note: this module, like UnicodeDammit, is very US/English-centric, and is practically useless for worldwide web scraping. For non-english pages, it is necessary to statistically estimate the codepage and language of each page segment, and then try to normalize each segment to unicode.
- m_mueller 12y ago
  The way I understand it from their examples, it's rather latin-written-languages-centric, no? Could you give an example where it doesn't work with a romanized language? If not, then I'd hardly call that English-centric and practically useless worldwide.
  - unicodedammit 12y ago
    from __init__.guess_bytes(): "This is not a magic bullet. If the bytes are coming from some MySQL database with the "character set" set to ISO Elbonian, this won't figure it out. Perhaps more relevantly, this currently doesn't try East Asian encodings." The world is a very large place, there are many codepages in use besides latin-1 and "ISO Elbonian". All central european contries use latin-2 (1250), or cyrillic codepage (1251). Since they are all single-byte codepages, they cannot be detected by try: convert() catch: try_another_codepage() and must be distinguished statistically. LTR/RTL language and asian encoding detection is even worse. https://en.wikipedia.org/wiki/Code_page https://en.wikipedia.org/wiki/Code_page https://en.wikipedia.org/wiki/Windows_code_pages https://en.wikipedia.org/wiki/Windows_code_pages Another python Unicode conversion module which is slightly less US/English-centric: https://github.com/buriy/python-readability https://github.com/buriy/python-readability
    - rspeer 12y ago
      I would like to emphasize that "guess_bytes" is not what ftfy is about. It's there for convenience, and because the command-line version can't work without it. But the main "fix_text" function isn't about guessing the encodings of bytes. Not all text arrives in the form of unmarked bytes. HTTP gives you bytes marked with an encoding. JSON straight up gives you Unicode. Once you have that Unicode, you might notice problems with it like mojibake, and ftfy is designed for fixing that. Like you say, encoding detection has to be done statistically. That's a great goal for another project (man, I wish chardet could do it), but once statistics get in there, it would be completely impossible to get a false positive rate as low as ftfy has.
  - dbfclark 12y ago
teddyh 12y ago
I can’t help but think that this merely gives people the excuse they need for not understanding this “Things-that-are-not-ASCII” problem. Using this library is a desperate attempt to have a just-fix-it function, but it can never cover all cases, and will inevitably corrupt data. To use this library is to remain an ASCII neanderthal, ignorant of the modern world and the difference of text, bytes and encodings. Let me explain in some detail why this library is not a good thing: In an ideal world, you would know what encoding bytes are in and could therefore decode them explicitly using the known correct encoding, and this library would be redundant. If instead, as is often the case in the real world, the coding is unknown, there exists the question of how to resolve the numerous ambiguities which result. A library such as this would have to guess what encoding to use in each specific instance, and the choices it ideally should make are extremely dependent on the circumstances and even the immediate context. As it is, the library is hard-coded with some specific algorithms to choose some encodings over others, and if those assumptions does not match your use case exactly, the library will corrupt your data. A much better solution would perhaps involve a machine learning solution to the problem, and having the library be trained to deduce the probable encodings from a large set of example data from each user’s individual use case. Even these will occasionally be wrong, but at least it would be the best we could do with unknown encodings without resorting to manual processing. However, a one-size-fits-all “solution” such as this is merely giving people a further excuse to keep not caring about encodings, to pretend that encodings can be “detected”, and that there exists such a thing as “plain text”.
- rspeer 12y ago
  I think you're criticizing the wrong library. ftfy isn't about encoding detection. Should I take guess_bytes out of the documentation to stop giving that impression? It's the library you use when the data you get has already been decoded incorrectly. The user of ftfy cares about encodings, but gets data from sources that don't. And in no practical sense does it corrupt your data. I don't know where you got that idea from. It leaves good data alone. I will not say that false positives are nonexistent, but they are vanishingly rare -- see http://ftfy.readthedocs.org/en/latest/#accuracy http://ftfy.readthedocs.org/en/latest/#accuracy -- and they don't occur in "serious" data, they occur when people are screwing around with bizarre emoticons and stuff.
TazeTSchnitzel 12y ago
This will mangle text discussing mojibake.
sylvinus 12y ago
The last testimonial in the README gave me a good laugh :)
kev009 12y ago
Had a dataset where this was the case.. old devs used Windows, I'm not sure what the DB encoding was set as when they did imports, etc. I've been putting off fixing it because it's just a PITA to deal with. But I built a sanitizer in a couple hours with this lib, and it seems to work pretty well. The only unexpected thing is that it converts the ordinal indicator º to o in addresses. Luckily there are only a handful I need to fix.
- dbfclark 12y ago
  This is an impact of the default Unicode normalization, which is set to NFKC. This normalization is lossy for things like the ordinal indicator and trademark symbol; if you'd like to keep the ordinal indicator unchanged, use NFC normalization: >>> print ftfy.fix_text(u'ordinal indicator º to o in addresses.') ordinal indicator o to o in addresses. >>> print ftfy.fix_text(u'ordinal indicator º to o in addresses.',normalization='NFC') ordinal indicator º to o in addresses.
lsferreira42 12y ago
Dude, i'll pay you a beer if you want, this project is fucking awsome!

101 points by rspeer 12y ago

tdumitrescu 12y ago

Wow, recruiting pitch at the top of the README. Someone make Adblock for Github.

artursapek 12y ago
I think the least we can give in return to companies that open-source useful stuff like this is look the other way when they self-promote a little bit.
jjgreen 12y ago
I clicked on the careers link and would seriously consider working for a company with such an enlightened attitude to open source, chill! (In fact in the wrong country, but nevertheless...)

meowface 12y ago

This is a great project because a lot of work has been put into finding solutions to many different edge cases, and it "just works".

rwg 12y ago

It even successfully un-mangles the shipping label from Ode to a Shipping Label: https://www.facebook.com/cmb/posts/619241744770551:0 https://www.facebook.com/cmb/posts/619241744770551:0 >>> ftfy.fix_text('L&amp;Atilde;&amp;sup3;pez') 'López' Bravo!

unicodedammit 12y ago

Very common problem in web scraping, for example forum site might contain a mix of MacRoman, Windows codepage and various european codepages in a single page (yes, even in 2014!). Seems like a more advanced version of UnicodeDammit module ( http://www.crummy.com/software/BeautifulSoup/bs4/doc/#unicode-dammit http://www.crummy.com/software/BeautifulSoup/bs4/doc/#unicod... ). Note: this module, like UnicodeDammit, is very US/English-centric, and is practically useless for worldwide web scraping. For non-english pages, it is necessary to statistically estimate the codepage and language of each page segment, and then try to normalize each segment to unicode.

m_mueller 12y ago
The way I understand it from their examples, it's rather latin-written-languages-centric, no? Could you give an example where it doesn't work with a romanized language? If not, then I'd hardly call that English-centric and practically useless worldwide.
- unicodedammit 12y ago
  from __init__.guess_bytes(): "This is not a magic bullet. If the bytes are coming from some MySQL database with the "character set" set to ISO Elbonian, this won't figure it out. Perhaps more relevantly, this currently doesn't try East Asian encodings." The world is a very large place, there are many codepages in use besides latin-1 and "ISO Elbonian". All central european contries use latin-2 (1250), or cyrillic codepage (1251). Since they are all single-byte codepages, they cannot be detected by try: convert() catch: try_another_codepage() and must be distinguished statistically. LTR/RTL language and asian encoding detection is even worse. https://en.wikipedia.org/wiki/Code_page https://en.wikipedia.org/wiki/Code_page https://en.wikipedia.org/wiki/Windows_code_pages https://en.wikipedia.org/wiki/Windows_code_pages Another python Unicode conversion module which is slightly less US/English-centric: https://github.com/buriy/python-readability https://github.com/buriy/python-readability
  - rspeer 12y ago
    I would like to emphasize that "guess_bytes" is not what ftfy is about. It's there for convenience, and because the command-line version can't work without it. But the main "fix_text" function isn't about guessing the encodings of bytes. Not all text arrives in the form of unmarked bytes. HTTP gives you bytes marked with an encoding. JSON straight up gives you Unicode. Once you have that Unicode, you might notice problems with it like mojibake, and ftfy is designed for fixing that. Like you say, encoding detection has to be done statistically. That's a great goal for another project (man, I wish chardet could do it), but once statistics get in there, it would be completely impossible to get a false positive rate as low as ftfy has.
- dbfclark 12y ago

teddyh 12y ago

I can’t help but think that this merely gives people the excuse they need for not understanding this “Things-that-are-not-ASCII” problem. Using this library is a desperate attempt to have a just-fix-it function, but it can never cover all cases, and will inevitably corrupt data. To use this library is to remain an ASCII neanderthal, ignorant of the modern world and the difference of text, bytes and encodings. Let me explain in some detail why this library is not a good thing: In an ideal world, you would know what encoding bytes are in and could therefore decode them explicitly using the known correct encoding, and this library would be redundant. If instead, as is often the case in the real world, the coding is unknown, there exists the question of how to resolve the numerous ambiguities which result. A library such as this would have to guess what encoding to use in each specific instance, and the choices it ideally should make are extremely dependent on the circumstances and even the immediate context. As it is, the library is hard-coded with some specific algorithms to choose some encodings over others, and if those assumptions does not match your use case exactly, the library will corrupt your data. A much better solution would perhaps involve a machine learning solution to the problem, and having the library be trained to deduce the probable encodings from a large set of example data from each user’s individual use case. Even these will occasionally be wrong, but at least it would be the best we could do with unknown encodings without resorting to manual processing. However, a one-size-fits-all “solution” such as this is merely giving people a further excuse to keep not caring about encodings, to pretend that encodings can be “detected”, and that there exists such a thing as “plain text”.

rspeer 12y ago
I think you're criticizing the wrong library. ftfy isn't about encoding detection. Should I take guess_bytes out of the documentation to stop giving that impression? It's the library you use when the data you get has already been decoded incorrectly. The user of ftfy cares about encodings, but gets data from sources that don't. And in no practical sense does it corrupt your data. I don't know where you got that idea from. It leaves good data alone. I will not say that false positives are nonexistent, but they are vanishingly rare -- see http://ftfy.readthedocs.org/en/latest/#accuracy http://ftfy.readthedocs.org/en/latest/#accuracy -- and they don't occur in "serious" data, they occur when people are screwing around with bizarre emoticons and stuff.

TazeTSchnitzel 12y ago

This will mangle text discussing mojibake.

sylvinus 12y ago

The last testimonial in the README gave me a good laugh :)

kev009 12y ago

Had a dataset where this was the case.. old devs used Windows, I'm not sure what the DB encoding was set as when they did imports, etc. I've been putting off fixing it because it's just a PITA to deal with. But I built a sanitizer in a couple hours with this lib, and it seems to work pretty well. The only unexpected thing is that it converts the ordinal indicator º to o in addresses. Luckily there are only a handful I need to fix.

dbfclark 12y ago
This is an impact of the default Unicode normalization, which is set to NFKC. This normalization is lossy for things like the ordinal indicator and trademark symbol; if you'd like to keep the ordinal indicator unchanged, use NFC normalization: >>> print ftfy.fix_text(u'ordinal indicator º to o in addresses.') ordinal indicator o to o in addresses. >>> print ftfy.fix_text(u'ordinal indicator º to o in addresses.',normalization='NFC') ordinal indicator º to o in addresses.

lsferreira42 12y ago

Dude, i'll pay you a beer if you want, this project is fucking awsome!