4 ms·

Maybe you're right. Maybe there are applications where this really matters. All I can say is that I have never once had that problem since I started using Pyt

Maybe you're right. Maybe there are applications where this really matters. All I can say is that I have never once had that problem since I started using Python (circa version 2.4). It has always been fine. For example, when dealing with .csv files created with MS Excel, you get strange characters because Excel uses CP1252 encoding, not ASCII. They never tripped up my app in python 2.7. It happily passed them along. Even regular expressions worked. Python3 died repeatedly until I went through and forced everything to be bytestrings.

SketchySeaBeast 7y ago
That's funny as I had the opposite of experience - pulling non-ASCII characters into Python 2.7 caused me trouble - I ended up having to mark everything as Unicode to solve the issues.
prewett 7y ago
You're reading non-Unicode into strings that expect Unicode, and since there's no way to auto-detect CP1252, you get bad results. If you want to stay in CP1252 you should clearly use bytestrings. You could convert to Unicode, I expect that: text = file.read().decode('cp1252') would work (haven't tried it, though). I frequently work with CJK text, and Python 2 was a real pain. You had to do things one way to output to a terminal, but a different way to output to a pipe, and all your code had to remember to encode('utf8') every time you printed a string (unless you were using pipes, if I recall correctly). The situation is much better with Python 3, especially if you keep your files in UTF-8. Unfortunately, I think Windows is still stuck in their own world, so it might be harder to get UTF-8 by default. In Unix pretty much everything writes UTF-8 files by default, so everything has pretty much just worked for me (macOS/Linux).
nemetroid 7y ago
> Even regular expressions worked. Did you try doing regular expressions for non-ASCII characters? For example, given a file "latin1file" containing the string "L£", encoded in Latin-1, Python 2 will fail silently: $ python2 -c 'import re; s = open("latin1file").read(); print(re.search("L", s));' <_sre.SRE_Match object at 0x7fb992a44bf8> $ python2 -c 'import re; s = open("latin1file").read(); print(re.search("£", s));' None Python3 forces you to deal with the problem to handle the first case, but if you do, the second case will work as well: $ python3 -c 'import re; s = open("latin1file").read(); print(re.search("L", s));' Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python3.7/codecs.py", line 322, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 1: invalid start byte $ python3 -c 'import re; s = open("latin1file", encoding="latin1").read(); print(re.search("L", s));' <re.Match object; span=(0, 1), match='L'> $ python3 -c 'import re; s = open("latin1file", encoding="latin1").read(); print(re.search("£", s));' <re.Match object; span=(1, 2), match='£'>
- metalliqaz 7y ago
  I've never had to do that. Maybe because I live in the U.S.

SketchySeaBeast 7y ago

That's funny as I had the opposite of experience - pulling non-ASCII characters into Python 2.7 caused me trouble - I ended up having to mark everything as Unicode to solve the issues.

prewett 7y ago

You're reading non-Unicode into strings that expect Unicode, and since there's no way to auto-detect CP1252, you get bad results. If you want to stay in CP1252 you should clearly use bytestrings. You could convert to Unicode, I expect that: text = file.read().decode('cp1252') would work (haven't tried it, though). I frequently work with CJK text, and Python 2 was a real pain. You had to do things one way to output to a terminal, but a different way to output to a pipe, and all your code had to remember to encode('utf8') every time you printed a string (unless you were using pipes, if I recall correctly). The situation is much better with Python 3, especially if you keep your files in UTF-8. Unfortunately, I think Windows is still stuck in their own world, so it might be harder to get UTF-8 by default. In Unix pretty much everything writes UTF-8 files by default, so everything has pretty much just worked for me (macOS/Linux).

nemetroid 7y ago

> Even regular expressions worked. Did you try doing regular expressions for non-ASCII characters? For example, given a file "latin1file" containing the string "L£", encoded in Latin-1, Python 2 will fail silently: $ python2 -c 'import re; s = open("latin1file").read(); print(re.search("L", s));' <_sre.SRE_Match object at 0x7fb992a44bf8> $ python2 -c 'import re; s = open("latin1file").read(); print(re.search("£", s));' None Python3 forces you to deal with the problem to handle the first case, but if you do, the second case will work as well: $ python3 -c 'import re; s = open("latin1file").read(); print(re.search("L", s));' Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python3.7/codecs.py", line 322, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 1: invalid start byte $ python3 -c 'import re; s = open("latin1file", encoding="latin1").read(); print(re.search("L", s));' <re.Match object; span=(0, 1), match='L'> $ python3 -c 'import re; s = open("latin1file", encoding="latin1").read(); print(re.search("£", s));' <re.Match object; span=(1, 2), match='£'>

metalliqaz 7y ago
I've never had to do that. Maybe because I live in the U.S.