3 ms·

in my experience, most people want bytestrings most of the time, and the main effect of changing the str type has been to drive a lot of traffic to SO when they

in my experience, most people want bytestrings most of the time, and the main effect of changing the str type has been to drive a lot of traffic to SO when they have encoding errors trying to deal with strings that used to "just work" in 2.7

beagle3 7y ago
The thing about "just works" in 2.7 is that it does ... until it doesn't, and when it doesn't, it's often way harder to resolve. If you're only using the original ASCII (ordinals 0-127), all is well, and the py3 distinction between strings and bytes seems to get in the way. But as soon as you have a single character in your input or data which is not ASCII, a lot of things stop working and it isn't even clear why, or how to fix it -- often, the error comes from a library-within-a-library-within-a-library, which did some manipulation that (wrongly) assumed some encoding or its properties.
- metalliqaz 7y ago
  Maybe you're right. Maybe there are applications where this really matters. All I can say is that I have never once had that problem since I started using Python (circa version 2.4). It has always been fine. For example, when dealing with .csv files created with MS Excel, you get strange characters because Excel uses CP1252 encoding, not ASCII. They never tripped up my app in python 2.7. It happily passed them along. Even regular expressions worked. Python3 died repeatedly until I went through and forced everything to be bytestrings.
  - SketchySeaBeast 7y ago
    That's funny as I had the opposite of experience - pulling non-ASCII characters into Python 2.7 caused me trouble - I ended up having to mark everything as Unicode to solve the issues.
  - prewett 7y ago
    You're reading non-Unicode into strings that expect Unicode, and since there's no way to auto-detect CP1252, you get bad results. If you want to stay in CP1252 you should clearly use bytestrings. You could convert to Unicode, I expect that: text = file.read().decode('cp1252') would work (haven't tried it, though). I frequently work with CJK text, and Python 2 was a real pain. You had to do things one way to output to a terminal, but a different way to output to a pipe, and all your code had to remember to encode('utf8') every time you printed a string (unless you were using pipes, if I recall correctly). The situation is much better with Python 3, especially if you keep your files in UTF-8. Unfortunately, I think Windows is still stuck in their own world, so it might be harder to get UTF-8 by default. In Unix pretty much everything writes UTF-8 files by default, so everything has pretty much just worked for me (macOS/Linux).
  - nemetroid 7y ago
    > Even regular expressions worked. Did you try doing regular expressions for non-ASCII characters? For example, given a file "latin1file" containing the string "L£", encoded in Latin-1, Python 2 will fail silently: $ python2 -c 'import re; s = open("latin1file").read(); print(re.search("L", s));' <_sre.SRE_Match object at 0x7fb992a44bf8> $ python2 -c 'import re; s = open("latin1file").read(); print(re.search("£", s));' None Python3 forces you to deal with the problem to handle the first case, but if you do, the second case will work as well: $ python3 -c 'import re; s = open("latin1file").read(); print(re.search("L", s));' Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python3.7/codecs.py", line 322, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 1: invalid start byte $ python3 -c 'import re; s = open("latin1file", encoding="latin1").read(); print(re.search("L", s));' <re.Match object; span=(0, 1), match='L'> $ python3 -c 'import re; s = open("latin1file", encoding="latin1").read(); print(re.search("£", s));' <re.Match object; span=(1, 2), match='£'>
    - metalliqaz 7y ago
      I've never had to do that. Maybe because I live in the U.S.
pmart123 7y ago
I’m not quite sure about this. I think while python 3 has a tendency to blowup on IO, I still think that is preferable than more cryptic problems. Python 3 did seem to handicap working in bytes though, but I think that has improved in the last few versions with the path objects and str accessing Unicode text.

beagle3 7y ago

The thing about "just works" in 2.7 is that it does ... until it doesn't, and when it doesn't, it's often way harder to resolve. If you're only using the original ASCII (ordinals 0-127), all is well, and the py3 distinction between strings and bytes seems to get in the way. But as soon as you have a single character in your input or data which is not ASCII, a lot of things stop working and it isn't even clear why, or how to fix it -- often, the error comes from a library-within-a-library-within-a-library, which did some manipulation that (wrongly) assumed some encoding or its properties.

metalliqaz 7y ago
Maybe you're right. Maybe there are applications where this really matters. All I can say is that I have never once had that problem since I started using Python (circa version 2.4). It has always been fine. For example, when dealing with .csv files created with MS Excel, you get strange characters because Excel uses CP1252 encoding, not ASCII. They never tripped up my app in python 2.7. It happily passed them along. Even regular expressions worked. Python3 died repeatedly until I went through and forced everything to be bytestrings.
- SketchySeaBeast 7y ago
  That's funny as I had the opposite of experience - pulling non-ASCII characters into Python 2.7 caused me trouble - I ended up having to mark everything as Unicode to solve the issues.
- prewett 7y ago
  You're reading non-Unicode into strings that expect Unicode, and since there's no way to auto-detect CP1252, you get bad results. If you want to stay in CP1252 you should clearly use bytestrings. You could convert to Unicode, I expect that: text = file.read().decode('cp1252') would work (haven't tried it, though). I frequently work with CJK text, and Python 2 was a real pain. You had to do things one way to output to a terminal, but a different way to output to a pipe, and all your code had to remember to encode('utf8') every time you printed a string (unless you were using pipes, if I recall correctly). The situation is much better with Python 3, especially if you keep your files in UTF-8. Unfortunately, I think Windows is still stuck in their own world, so it might be harder to get UTF-8 by default. In Unix pretty much everything writes UTF-8 files by default, so everything has pretty much just worked for me (macOS/Linux).
- nemetroid 7y ago
  > Even regular expressions worked. Did you try doing regular expressions for non-ASCII characters? For example, given a file "latin1file" containing the string "L£", encoded in Latin-1, Python 2 will fail silently: $ python2 -c 'import re; s = open("latin1file").read(); print(re.search("L", s));' <_sre.SRE_Match object at 0x7fb992a44bf8> $ python2 -c 'import re; s = open("latin1file").read(); print(re.search("£", s));' None Python3 forces you to deal with the problem to handle the first case, but if you do, the second case will work as well: $ python3 -c 'import re; s = open("latin1file").read(); print(re.search("L", s));' Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python3.7/codecs.py", line 322, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 1: invalid start byte $ python3 -c 'import re; s = open("latin1file", encoding="latin1").read(); print(re.search("L", s));' <re.Match object; span=(0, 1), match='L'> $ python3 -c 'import re; s = open("latin1file", encoding="latin1").read(); print(re.search("£", s));' <re.Match object; span=(1, 2), match='£'>
  - metalliqaz 7y ago
    I've never had to do that. Maybe because I live in the U.S.

pmart123 7y ago

I’m not quite sure about this. I think while python 3 has a tendency to blowup on IO, I still think that is preferable than more cryptic problems. Python 3 did seem to handicap working in bytes though, but I think that has improved in the last few versions with the path objects and str accessing Unicode text.