4 ms·

One thing that amuses me given the number of complaints about the Python 3 string transition is how vastly better Python 3 is for working with bytes. The infras

One thing that amuses me given the number of complaints about the Python 3 string transition is how vastly better Python 3 is for working with bytes. The infrastructure available is light-years ahead of what Python 2 offered, precisely because it gave up on trying to also make bags of bytes be the default string type.

acdha 8y ago
Agreed. So many of the complaints leave me wanting to suggest that someone try it for longer than 15 minutes and see what they think then.
ak217 8y ago
Thank you for saying that. Working with strings and bytes in Python 3 is nothing short of a joy compared to the dodgy stuff Python 2 did. People who complain about the change are delusional.
- necovek 8y ago
  The only problem I have with Python3 strings/bytes handling is the fact that there are standard library functions which accept bytestrings in Py2 (regular "" strings), and Unicode strings in Py3 (again, regular "" strings in Py3). This has led to developers attempting to conflate the two distinctly different concepts and make APIs support both while behaving differently. A simple solution is there in plain sight: just use exclusively b"" and u"" strings for any code you wish to work in both Py2 and Py3, and forget about "". All and any libraries should be using those exclusively if they support both. Python3-only code should be using b"" and "" instead. One could consider this a design oversight in Python 3: the fact that the syntax is so similar elsewhere makes people want to run the same code in both, yet a core type is basically incompatible.
  - joshuamorton 8y ago
    u"" is a syntax error in python3 (or at least it was for a while, apparently its not anymore, that said...). The correct cross-platform solution is to do from __future__ import unicode_literals which makes python2 string literals unicode unless declared bytes. Then "" strings are always unicode and b"" strings are always bytes, no matter the language version.
    - ak217 8y ago
      > u"" is a syntax error in python3 This has not been the case since 2012. The last release of Python 3 for which this was the case reached end of life in February 2016. Please stop misinforming people.
      - deleted 8y ago
        [deleted]
      - Gorgor 8y ago
        While u"" is accepted in current Python 3, for some reason they ignored the raw unicode string ur"" which still is a syntax error in Python 3. So, unicode_literals is definitely preferable.
    - masklinn 8y ago
      > The correct cross-platform solution is to do It's absolutely not correct, because there are many APIs which take "native strings" across versions aka they take an `str` (byte string) in Python 2 and an `str` (unicode string) in Python 3. unicode_literals causes significantly more problems than they solve. The correct cross-platform solution (and the very reason why the `u` prefix was reintroduced after having initially been removed from Python 3) is in fact to use b"" for known byte strings, u"" for known text, and "" for "native strings" to interact with APIs which specifically need that.
marcosdumay 8y ago
Edit: I didn't try to handle files with Python for a long time. All of the following is stuff that can change as an ecosystem matures. All that may not hold anymore. It's better in some ways. But then, I gave up long ago on manipulating files with Python, because Python 3 simply decided that anything on my filesystem is utf-8 by default. Want to get an stream out of a file? Better have utf-8 content. Otherwise all the documentation will be buried 15 meters deep, outdated on some aspects and dependent on some not-released yet features. Want to name a file? Better have an utf-8 name. I gave-up on python before I was able to open a file with a name that doesn't conform to utf-8. Want to read a line at a time? Sorry, we only do that to text. Go get an utf-8 file. Want to match with a regular expression or parse it somehow? Sorry, we only do that to text. And so on.
- ak217 8y ago
  Perhaps your information is out of date? I think none of what you said is true. Maybe some of it was true in the past. > Want to get an stream out of a file? Better have utf-8 content. Otherwise all the documentation will be buried 15 meters deep, outdated on some aspects and dependent on some not-released yet features. f = open("myfile.txt", "r", encoding=ENCODING) > Want to name a file? Better have an utf-8 name. I gave-up on python before I was able to open a file with a name that doesn't conform to utf-8. export NAME="$(head -c 5 /dev/urandom)"; touch "$NAME"; python -c 'import os; open(os.environ["NAME"])' > Want to read a line at a time? Sorry, we only do that to text. Go get an utf-8 file. fh = open("foo", "rb"); lines=[l for l in fh] > Want to match with a regular expression or parse it somehow? Sorry, we only do that to text. import re; re.match(b"abc", b"abcdef")
  - marcosdumay 8y ago
    Yes, my experience is probably out of date.
hermitdev 8y ago
I haven't noticed much pain from the ASCII -> UNICODE migration from 2->3, but the one thing that still bothers me is that they did not update the CSV module to be transparent. In particular, the need to explicitly set the newline argument to `csv.reader`[0]. For me, dealing with a lot of data translation/extraction/loading work (ETL), this has been a big annoyance in migrating from 2 to 3. I've not really had many other issues porting from 2 -> 3 from my own code; issues usually arise from 3rd-party libs that are relied upon (especially if they utilize C/C++ extensions). DB libs have sometimes been problematic. IIRC, pysybase requires you to set the encoding of strings now (which wasn't required before). I use pysybase to talk to both Sybase & MS SQL (it talks the TDS protocol). [0] https://docs.python.org/3/library/csv.html#csv.reader https://docs.python.org/3/library/csv.html#csv.reader
- masklinn 8y ago
  I haven't noticed much pain from the ASCII -> UNICODE migration from 2->3, but the one thing that still bothers me is that they did not update the CSV module to be transparent. In particular, the need to explicitly set the newline argument to `csv.reader`[0]. For me, dealing with a lot of data translation/extraction/loading work (ETL), this has been a big annoyance in migrating from 2 to 3. ??? The Python 3 CSV module works on text only (unicode), and you seem to be misreading the note: the module does in fact do newline transparency: https://docs.python.org/3/library/csv.html#id3 https://docs.python.org/3/library/csv.html#id3 `newline=''` is to be specified on the file so it doesn't perform newline translation, because that would change the semantics of newlines in quoted strings: by default, `open` will universally translate newlines to `\n`, `newline=''` means it still does universal newlines (for purpose of iteration) but it doesn't do translation, returning newline sequences unchanged. The Python 2 CSV module only worked on byte strings, and magic bytes.

acdha 8y ago

Agreed. So many of the complaints leave me wanting to suggest that someone try it for longer than 15 minutes and see what they think then.

ak217 8y ago

Thank you for saying that. Working with strings and bytes in Python 3 is nothing short of a joy compared to the dodgy stuff Python 2 did. People who complain about the change are delusional.

necovek 8y ago
The only problem I have with Python3 strings/bytes handling is the fact that there are standard library functions which accept bytestrings in Py2 (regular "" strings), and Unicode strings in Py3 (again, regular "" strings in Py3). This has led to developers attempting to conflate the two distinctly different concepts and make APIs support both while behaving differently. A simple solution is there in plain sight: just use exclusively b"" and u"" strings for any code you wish to work in both Py2 and Py3, and forget about "". All and any libraries should be using those exclusively if they support both. Python3-only code should be using b"" and "" instead. One could consider this a design oversight in Python 3: the fact that the syntax is so similar elsewhere makes people want to run the same code in both, yet a core type is basically incompatible.
- joshuamorton 8y ago
  u"" is a syntax error in python3 (or at least it was for a while, apparently its not anymore, that said...). The correct cross-platform solution is to do from __future__ import unicode_literals which makes python2 string literals unicode unless declared bytes. Then "" strings are always unicode and b"" strings are always bytes, no matter the language version.
  - ak217 8y ago
    > u"" is a syntax error in python3 This has not been the case since 2012. The last release of Python 3 for which this was the case reached end of life in February 2016. Please stop misinforming people.
    - deleted 8y ago
      [deleted]
    - Gorgor 8y ago
      While u"" is accepted in current Python 3, for some reason they ignored the raw unicode string ur"" which still is a syntax error in Python 3. So, unicode_literals is definitely preferable.
  - masklinn 8y ago
    > The correct cross-platform solution is to do It's absolutely not correct, because there are many APIs which take "native strings" across versions aka they take an `str` (byte string) in Python 2 and an `str` (unicode string) in Python 3. unicode_literals causes significantly more problems than they solve. The correct cross-platform solution (and the very reason why the `u` prefix was reintroduced after having initially been removed from Python 3) is in fact to use b"" for known byte strings, u"" for known text, and "" for "native strings" to interact with APIs which specifically need that.

marcosdumay 8y ago

Edit: I didn't try to handle files with Python for a long time. All of the following is stuff that can change as an ecosystem matures. All that may not hold anymore. It's better in some ways. But then, I gave up long ago on manipulating files with Python, because Python 3 simply decided that anything on my filesystem is utf-8 by default. Want to get an stream out of a file? Better have utf-8 content. Otherwise all the documentation will be buried 15 meters deep, outdated on some aspects and dependent on some not-released yet features. Want to name a file? Better have an utf-8 name. I gave-up on python before I was able to open a file with a name that doesn't conform to utf-8. Want to read a line at a time? Sorry, we only do that to text. Go get an utf-8 file. Want to match with a regular expression or parse it somehow? Sorry, we only do that to text. And so on.

ak217 8y ago
Perhaps your information is out of date? I think none of what you said is true. Maybe some of it was true in the past. > Want to get an stream out of a file? Better have utf-8 content. Otherwise all the documentation will be buried 15 meters deep, outdated on some aspects and dependent on some not-released yet features. f = open("myfile.txt", "r", encoding=ENCODING) > Want to name a file? Better have an utf-8 name. I gave-up on python before I was able to open a file with a name that doesn't conform to utf-8. export NAME="$(head -c 5 /dev/urandom)"; touch "$NAME"; python -c 'import os; open(os.environ["NAME"])' > Want to read a line at a time? Sorry, we only do that to text. Go get an utf-8 file. fh = open("foo", "rb"); lines=[l for l in fh] > Want to match with a regular expression or parse it somehow? Sorry, we only do that to text. import re; re.match(b"abc", b"abcdef")
- marcosdumay 8y ago
  Yes, my experience is probably out of date.

hermitdev 8y ago

I haven't noticed much pain from the ASCII -> UNICODE migration from 2->3, but the one thing that still bothers me is that they did not update the CSV module to be transparent. In particular, the need to explicitly set the newline argument to `csv.reader`[0]. For me, dealing with a lot of data translation/extraction/loading work (ETL), this has been a big annoyance in migrating from 2 to 3. I've not really had many other issues porting from 2 -> 3 from my own code; issues usually arise from 3rd-party libs that are relied upon (especially if they utilize C/C++ extensions). DB libs have sometimes been problematic. IIRC, pysybase requires you to set the encoding of strings now (which wasn't required before). I use pysybase to talk to both Sybase & MS SQL (it talks the TDS protocol). [0] https://docs.python.org/3/library/csv.html#csv.reader https://docs.python.org/3/library/csv.html#csv.reader

masklinn 8y ago
I haven't noticed much pain from the ASCII -> UNICODE migration from 2->3, but the one thing that still bothers me is that they did not update the CSV module to be transparent. In particular, the need to explicitly set the newline argument to `csv.reader`[0]. For me, dealing with a lot of data translation/extraction/loading work (ETL), this has been a big annoyance in migrating from 2 to 3. ??? The Python 3 CSV module works on text only (unicode), and you seem to be misreading the note: the module does in fact do newline transparency: https://docs.python.org/3/library/csv.html#id3 https://docs.python.org/3/library/csv.html#id3 `newline=''` is to be specified on the file so it doesn't perform newline translation, because that would change the semantics of newlines in quoted strings: by default, `open` will universally translate newlines to `\n`, `newline=''` means it still does universal newlines (for purpose of iteration) but it doesn't do translation, returning newline sequences unchanged. The Python 2 CSV module only worked on byte strings, and magic bytes.