2 ms·
You're reading non-Unicode into strings that expect Unicode, and since there's no way to auto-detect CP1252, you get bad results. If you want to stay in CP1252
by prewett 7y ago
You're reading non-Unicode into strings that expect Unicode, and since there's no way to auto-detect CP1252, you get bad results. If you want to stay in CP1252 you should clearly use bytestrings. You could convert to Unicode, I expect that:
text = file.read().decode('cp1252')
would work (haven't tried it, though).
I frequently work with CJK text, and Python 2 was a real pain. You had to do things one way to output to a terminal, but a different way to output to a pipe, and all your code had to remember to encode('utf8') every time you printed a string (unless you were using pipes, if I recall correctly). The situation is much better with Python 3, especially if you keep your files in UTF-8. Unfortunately, I think Windows is still stuck in their own world, so it might be harder to get UTF-8 by default. In Unix pretty much everything writes UTF-8 files by default, so everything has pretty much just worked for me (macOS/Linux).