3 ms·
Had a dataset where this was the case.. old devs used Windows, I'm not sure what the DB encoding was set as when they did imports, etc. I've been putting off f
by kev009 12y ago
Had a dataset where this was the case.. old devs used Windows, I'm not sure what the DB encoding was set as when they did imports, etc. I've been putting off fixing it because it's just a PITA to deal with.
But I built a sanitizer in a couple hours with this lib, and it seems to work pretty well.
The only unexpected thing is that it converts the ordinal indicator º to o in addresses. Luckily there are only a handful I need to fix.
- dbfclark 12y agoThis is an impact of the default Unicode normalization, which is set to NFKC. This normalization is lossy for things like the ordinal indicator and trademark symbol; if you'd like to keep the ordinal indicator unchanged, use NFC normalization: >>> print ftfy.fix_text(u'ordinal indicator º to o in addresses.') ordinal indicator o to o in addresses. >>> print ftfy.fix_text(u'ordinal indicator º to o in addresses.',normalization='NFC') ordinal indicator º to o in addresses.