11 ms·

How Python 3 Should Have Worked (2012)

101 points by denzil_correa 13y ago

winstonian 13y ago
One of the top posts on the Python Reddit today: http://python-future.org/ http://python-future.org/
jfaucett 13y ago
This strategy makes a lot of sense to me. I really like the "deprecated warnings" to "explicit failure" transitions, IMO this works really well, though I only have experience with it at a library level using semver. So cant say much about point #1, except that this is essentially what you're doing when you test agains xlib-head branch. Aaron implies that this approach was not taken with Python (non-py guy here), could someone tell me what the reasoning was behind that? To much legacy code in Py2 code base, just wanting to start from a clean slate, or what? EDIT: (to add another question :) could you efficiently accomplish what Aaron is talking about, what would be the best way to go about this? @pak would you really have to load 2 stdlibs, is there no (efficient) way around syntax errors?
- aston 13y ago
  There are pieces of Python 3 that are syntax errors in Python 2. And there are Python 2-isms that are valid syntax in Python 3 but have a different interpretation. It's not as simple as importing certain features (which creates a sort of language version hybrid). The idea to allow one project to switch between Python 2 and Python 3 for individual files is more interesting, but practically speaking would lead to sort of a mess.
  - pak 13y ago
    >The idea to allow one project to switch between Python 2 and 3 for individual files Yes, I believe key parts of the standard object model changed between the two (e.g. strings vs bytes, many of the magic methods and operators) making this nearly impossible. Every time objects would pass back and forth, they'd have to be converted, which is wasteful and bug-prone (and this is a whole mess of library code that the python3 guys probably did not want to write). You'd also need to load two different standard libraries, which would waste memory. You only need to scan through the upgrade feature list to see how hard intercompatibility would have been. http://docs.python.org/3.0/whatsnew/3.0.html http://docs.python.org/3.0/whatsnew/3.0.html Although I totally agree with Aaron that this would have allowed people to actually use Python 3 without fear, anything short of forcing the entire program and all of its modules to run in v3 mode as opposed to v2 mode would have been a disaster from a reliability and technical design standpoint. And that's closer to how things actually went down with 2to3, etc.
    - jfaucett 13y ago
      thanks for that link, now I see the problems Text Data vs. Unicode alone would be an enormous overhaul. Though the syntax changes dont seem that problematic.
      - bskap 13y ago
        They aren't. And there are automatic tools for converting between them (2to3 and 3to2), along with those __future__ imports that Aaron mentioned: doing "from __future__ import print_function, unicode_literals, absolute_import, division" would give you most of the Python 3 syntax changes in Python 2. The "everything expects bytes" to "everything expects text" change is the biggest hurdle for a lot of projects.
- pessimizer 13y ago
  This link is posted from the [dead] winstonian: http://python-future.org/ http://python-future.org/
  - winstonian 13y ago
    Why am I listed as [dead]?
edanm 13y ago
I think the strategy Aaron talks about makes a lot of sense. I especially like the idea of simply shipping future interpereters that can work with both 2.x and 3.x code. Seriously, it makes it even more dead-simple to get started with Python 3. We can extend Aaron's ideas to even more radical ideas, for example, instead of allowing Python 3 and 2 code to be mixed on a per-file basis, allow it to be mixed on a per-function basis. In fact, allow running Python 3 code, and drop in "backwards incompatible" blocks inside of a function to let you program things that will be backwards-compatible. In other words, let people program in Python 3 as much as they want, but allow them a way to use libraries that only support Python 2 without making a mess. I'm not saying this will be easy at all, but it will definitely make Py3k adoption actually happen. On the meta level, I'm really glad people are now discussing how to get the Python 3 rollout happening, because we really are dangerously close to having a "dead" language in Python if nothing changes.
overgard 13y ago
I agree with this 100%. Python 3 is a disaster. The problem is they made the entire thing out to be a Big Deal, but they didn't really offer any compelling reason to upgrade. I mean, the unicode is... kinda better, and iterators are a bit improved, but couldn't those things have been point releases? 2.8? They basically said that python 3 was a new language, and then offered no significant reason you should use this new language. So we all kept using actual python, quirks and all. To me, as a python developer, Python 3 is a failed fork. Harsh but true. IMO, if they were going to do that sort of thing, they should have had at least one killer feature. Like maybe if python 3 had been based off pypy they could be saying "Look! We're 5x faster! Want to upgrade now?". That would have been compelling. But their message was: "we cleaned up some stuff that most of you don't care about, and broke a bunch of things". Think about pitching that sort of upgrade to your boss. "Well it doesn't solve any of our problems, and it creates a ton of new ones, but it's the right thing to do because it makes some code slightly cleaner arguably! Convinced yet?" If I were in charge of python, I would do this: announce python 4, have it be based on the pypy interpreter, and keep compatibility with python 2 the language while reforming the C extension APIs to be more future proof (for getting rid of the GIL and so on). (Or maybe get rid of them entirely and just have people use CFFI.)
- d0mine 13y ago
  Unrelated: You do know that pypy also uses GIL. CFFI is to call C functions from Python. It won't allow you to run C extensions for CPython (something should implement Python C API).
  - overgard 13y ago
    I do know that; although they've done some interesting work with STM recently and hopefully that works out. I remember in the past hearing about someone actually submitting a patch to cpython remove the GIL about 10 years ago, but it was rejected because it made the language about 2-10x slower. IMO, that decision was INSANE. I have a 16 core machine right now and it can only really use 1/16th of it. Nobody uses python for it's speed, and people that need it are probably using C extension libraries like numpy, so if you were to cut the language's speed in half for a short term I'm going to say that pretty much nobody would care, especially if it were to solve a problem that everybody hates. Ruby is already about 10x slower than python on most of tasks, and it's just not a big deal. GVR is optimizing for the wrong thing.
fear91 13y ago
It's also really off putting for beginners that try to learn the language. Python 3 is being served as the main download when you search for it. And while you search for tutorials, most of them are in python 2 - and 80% of them DO NOT state whether they are for python 2 or python 3 ( because most were made during the python 2 times? ). So people try to learn to code with python 3.x and get frustrated because simplest things don't work. They should seriously rethink the whole 3.x thing. I must say, introducing it probably did more harm than good for the future of this language.
- maxerickson 13y ago
  This is the major result when you Google "python download": http://www.python.org/getit/ http://www.python.org/getit/ It is the same as this page: http://www.python.org/download/ http://www.python.org/download/ That page gives fairly equal weight to the two versions (I guess that could have changed over time). Edit: It might make sense to have a warning about matching the interpreter version up with the tutorial, but clear wording for it is not obvious to me.
  - fear91 13y ago
    People tend to click first links with way higher frequency than those located below. Additionally, when a beginner sees this page, he sees two versions - of which one is 3 and one is 2 - I think most of people will choose the "newer" ( newer = better? ) version because they don't really know about the differences between the two. The second part of the problem is the fragmentation of tutorials on other websites - which can't be fixed by changing the download page.
  - briancurtin 13y ago
    The reason for the "getit" page was, at least at one point, because the "download" page was blocked throughout China.
nostrademons 13y ago
This misses the point of why Python 3 was invented: Unicode. Python 2's string handling is broken in the presence of unicode characters, often leading to subtle errors that wouldn't cause exceptions until far away from the place where the error was introduced, and oftentimes didn't produce exceptions at all, just wrong data. Strings were defined as sequences of bytes, and then provided a .decode method to convert them to a unicode object that stores them as a sequence of codepoints. The problem was that a large number of libraries (including all of Aaron's that I've looked at) used str as their internal string type, which meant they were storing a sequence of bytes in an arbitrary encoding but not storing the encoding along with it. If you pass such a library a string in a different encoding, it will happily store it, manipulate it, and concatenate it with other strings. If you pass such a library multiple strings in multiple encodings (like, for example, if you're pulling data from multiple webpages), you will get garbage data that can't be decoded in any codec. Python 3 changes this so that str stores unicode codepoints and there's a separate 'bytes' type for uninterpreted bytes, and you are supposed to decode your bytes into strings at system boundaries. This is recommended software engineering practice for anyone who builds large systems that have to interact with foreign-language text; however, a large number of Python developers work in English-only environments where anything they receive will automatically be ASCII. They've never tried to track down subtly broken encoding issues; for them, the decode step is extra busywork that seems pointless. The reason the Python2->3 transition has been so painful is that it involves a whole language ecosystem fixing bugs in their software, but the bugs are subtle enough that the vast majority of people doing the work will never have encountered them. You can't just use the "from __future__ import python3_unicode" support because this is a change to the semantics of an existing language feature. In Python2, a string is a sequence of bytes. In Python3, a string is a sequence of unicode codepoints. What happens when a Python3 program calls a Python2 library with a string object? Do you try to auto-convert the strings? You can't, really, because strings in Python2 don't specify their encoding; you have no way of knowing which codec the Python2 library meant, because chances are they didn't think about it. The other major change in Python 3 - iterators everywhere - is similar, and it's a recognition that an increasingly large proportion of the programming ecosystem lives in a world where async operation is important and many concurrent activities may be happening at once. And I'm really glad to see Python willing to take on these challenges even with 5 years of short-term pain, because it shows a commitment to keeping Python relevant for the issues that 21st-century programmers will face. An increasing number of software platforms will have to deal with non-English text; an increasing number will need to handle concurrent, event-based environments. Without these changes Python would basically cede these areas to languages like Go or Javascript that have their unicode story straight and are well-adapted to async programming.
philh 13y ago
> 3. In Python 2.c, warnings begun being issued when you tried to use the old way, explaining you needed to change or your code would stop working. > 4. In Python 2.d, it actually did stop working. I'm curious, what are some examples here? The `from __future__`s that I recall offhand are `print_function`, `division` (still need to be explicit) and the old `with_statement` (incompatible code broke as soon as this became default). The other old way of doing things that I can think of offhand is `catch Exception, e`, which has been replaced with `catch Exception as e`, but not removed.
- sepeth 13y ago
  there is `unicode_literals` too. For the full list, see http://docs.python.org/2/library/__future__.html http://docs.python.org/2/library/__future__.html And there is one forgotten in that list: from __future__ import braces
  - philh 13y ago
    So it looks like in Aaron's list, 2.a == 2.c and 2.b == 2.d, and the process has been completed only three times, and two of those times were just adding keywords.
glynjackson 13y ago
quote from python.org: "New to Python or choosing between Python 2 and Python 3? Read Python 2 or Python 3." "Which version you ought to use is mostly dependent on what you want to get done." No other language I have ever worked with speaks like this. "New to Rudy? You could use version 2.1 or why not try version 1.0, it still works ok" Python.org treats version 2 and 3 as completely different things, newbies to the language like myself don't see it as a update to Python 2.x because thats not how it's sold to us.
jbeja 13y ago
I think the only people nthat hate python 3 is the one that have worked with it for several years, newbies wouldn't even care a little.
codecondo 13y ago
This aaron guy likes to talk a lot of shit so it seems

101 points by denzil_correa 13y ago

One of the top posts on the Python Reddit today: http://python-future.org/ http://python-future.org/

This strategy makes a lot of sense to me. I really like the "deprecated warnings" to "explicit failure" transitions, IMO this works really well, though I only have experience with it at a library level using semver. So cant say much about point #1, except that this is essentially what you're doing when you test agains xlib-head branch. Aaron implies that this approach was not taken with Python (non-py guy here), could someone tell me what the reasoning was behind that? To much legacy code in Py2 code base, just wanting to start from a clean slate, or what? EDIT: (to add another question :) could you efficiently accomplish what Aaron is talking about, what would be the best way to go about this? @pak would you really have to load 2 stdlibs, is there no (efficient) way around syntax errors?

aston 13y ago
There are pieces of Python 3 that are syntax errors in Python 2. And there are Python 2-isms that are valid syntax in Python 3 but have a different interpretation. It's not as simple as importing certain features (which creates a sort of language version hybrid). The idea to allow one project to switch between Python 2 and Python 3 for individual files is more interesting, but practically speaking would lead to sort of a mess.
- pak 13y ago
  >The idea to allow one project to switch between Python 2 and 3 for individual files Yes, I believe key parts of the standard object model changed between the two (e.g. strings vs bytes, many of the magic methods and operators) making this nearly impossible. Every time objects would pass back and forth, they'd have to be converted, which is wasteful and bug-prone (and this is a whole mess of library code that the python3 guys probably did not want to write). You'd also need to load two different standard libraries, which would waste memory. You only need to scan through the upgrade feature list to see how hard intercompatibility would have been. http://docs.python.org/3.0/whatsnew/3.0.html http://docs.python.org/3.0/whatsnew/3.0.html Although I totally agree with Aaron that this would have allowed people to actually use Python 3 without fear, anything short of forcing the entire program and all of its modules to run in v3 mode as opposed to v2 mode would have been a disaster from a reliability and technical design standpoint. And that's closer to how things actually went down with 2to3, etc.
  - jfaucett 13y ago
    thanks for that link, now I see the problems Text Data vs. Unicode alone would be an enormous overhaul. Though the syntax changes dont seem that problematic.
    - bskap 13y ago
      They aren't. And there are automatic tools for converting between them (2to3 and 3to2), along with those __future__ imports that Aaron mentioned: doing "from __future__ import print_function, unicode_literals, absolute_import, division" would give you most of the Python 3 syntax changes in Python 2. The "everything expects bytes" to "everything expects text" change is the biggest hurdle for a lot of projects.
pessimizer 13y ago
This link is posted from the [dead] winstonian: http://python-future.org/ http://python-future.org/
- winstonian 13y ago
  Why am I listed as [dead]?

edanm 13y ago

I think the strategy Aaron talks about makes a lot of sense. I especially like the idea of simply shipping future interpereters that can work with both 2.x and 3.x code. Seriously, it makes it even more dead-simple to get started with Python 3. We can extend Aaron's ideas to even more radical ideas, for example, instead of allowing Python 3 and 2 code to be mixed on a per-file basis, allow it to be mixed on a per-function basis. In fact, allow running Python 3 code, and drop in "backwards incompatible" blocks inside of a function to let you program things that will be backwards-compatible. In other words, let people program in Python 3 as much as they want, but allow them a way to use libraries that only support Python 2 without making a mess. I'm not saying this will be easy at all, but it will definitely make Py3k adoption actually happen. On the meta level, I'm really glad people are now discussing how to get the Python 3 rollout happening, because we really are dangerously close to having a "dead" language in Python if nothing changes.

overgard 13y ago

I agree with this 100%. Python 3 is a disaster. The problem is they made the entire thing out to be a Big Deal, but they didn't really offer any compelling reason to upgrade. I mean, the unicode is... kinda better, and iterators are a bit improved, but couldn't those things have been point releases? 2.8? They basically said that python 3 was a new language, and then offered no significant reason you should use this new language. So we all kept using actual python, quirks and all. To me, as a python developer, Python 3 is a failed fork. Harsh but true. IMO, if they were going to do that sort of thing, they should have had at least one killer feature. Like maybe if python 3 had been based off pypy they could be saying "Look! We're 5x faster! Want to upgrade now?". That would have been compelling. But their message was: "we cleaned up some stuff that most of you don't care about, and broke a bunch of things". Think about pitching that sort of upgrade to your boss. "Well it doesn't solve any of our problems, and it creates a ton of new ones, but it's the right thing to do because it makes some code slightly cleaner arguably! Convinced yet?" If I were in charge of python, I would do this: announce python 4, have it be based on the pypy interpreter, and keep compatibility with python 2 the language while reforming the C extension APIs to be more future proof (for getting rid of the GIL and so on). (Or maybe get rid of them entirely and just have people use CFFI.)

d0mine 13y ago
Unrelated: You do know that pypy also uses GIL. CFFI is to call C functions from Python. It won't allow you to run C extensions for CPython (something should implement Python C API).
- overgard 13y ago
  I do know that; although they've done some interesting work with STM recently and hopefully that works out. I remember in the past hearing about someone actually submitting a patch to cpython remove the GIL about 10 years ago, but it was rejected because it made the language about 2-10x slower. IMO, that decision was INSANE. I have a 16 core machine right now and it can only really use 1/16th of it. Nobody uses python for it's speed, and people that need it are probably using C extension libraries like numpy, so if you were to cut the language's speed in half for a short term I'm going to say that pretty much nobody would care, especially if it were to solve a problem that everybody hates. Ruby is already about 10x slower than python on most of tasks, and it's just not a big deal. GVR is optimizing for the wrong thing.

fear91 13y ago

It's also really off putting for beginners that try to learn the language. Python 3 is being served as the main download when you search for it. And while you search for tutorials, most of them are in python 2 - and 80% of them DO NOT state whether they are for python 2 or python 3 ( because most were made during the python 2 times? ). So people try to learn to code with python 3.x and get frustrated because simplest things don't work. They should seriously rethink the whole 3.x thing. I must say, introducing it probably did more harm than good for the future of this language.

maxerickson 13y ago
This is the major result when you Google "python download": http://www.python.org/getit/ http://www.python.org/getit/ It is the same as this page: http://www.python.org/download/ http://www.python.org/download/ That page gives fairly equal weight to the two versions (I guess that could have changed over time). Edit: It might make sense to have a warning about matching the interpreter version up with the tutorial, but clear wording for it is not obvious to me.
- fear91 13y ago
  People tend to click first links with way higher frequency than those located below. Additionally, when a beginner sees this page, he sees two versions - of which one is 3 and one is 2 - I think most of people will choose the "newer" ( newer = better? ) version because they don't really know about the differences between the two. The second part of the problem is the fragmentation of tutorials on other websites - which can't be fixed by changing the download page.
- briancurtin 13y ago
  The reason for the "getit" page was, at least at one point, because the "download" page was blocked throughout China.

nostrademons 13y ago

This misses the point of why Python 3 was invented: Unicode. Python 2's string handling is broken in the presence of unicode characters, often leading to subtle errors that wouldn't cause exceptions until far away from the place where the error was introduced, and oftentimes didn't produce exceptions at all, just wrong data. Strings were defined as sequences of bytes, and then provided a .decode method to convert them to a unicode object that stores them as a sequence of codepoints. The problem was that a large number of libraries (including all of Aaron's that I've looked at) used str as their internal string type, which meant they were storing a sequence of bytes in an arbitrary encoding but not storing the encoding along with it. If you pass such a library a string in a different encoding, it will happily store it, manipulate it, and concatenate it with other strings. If you pass such a library multiple strings in multiple encodings (like, for example, if you're pulling data from multiple webpages), you will get garbage data that can't be decoded in any codec. Python 3 changes this so that str stores unicode codepoints and there's a separate 'bytes' type for uninterpreted bytes, and you are supposed to decode your bytes into strings at system boundaries. This is recommended software engineering practice for anyone who builds large systems that have to interact with foreign-language text; however, a large number of Python developers work in English-only environments where anything they receive will automatically be ASCII. They've never tried to track down subtly broken encoding issues; for them, the decode step is extra busywork that seems pointless. The reason the Python2->3 transition has been so painful is that it involves a whole language ecosystem fixing bugs in their software, but the bugs are subtle enough that the vast majority of people doing the work will never have encountered them. You can't just use the "from __future__ import python3_unicode" support because this is a change to the semantics of an existing language feature. In Python2, a string is a sequence of bytes. In Python3, a string is a sequence of unicode codepoints. What happens when a Python3 program calls a Python2 library with a string object? Do you try to auto-convert the strings? You can't, really, because strings in Python2 don't specify their encoding; you have no way of knowing which codec the Python2 library meant, because chances are they didn't think about it. The other major change in Python 3 - iterators everywhere - is similar, and it's a recognition that an increasingly large proportion of the programming ecosystem lives in a world where async operation is important and many concurrent activities may be happening at once. And I'm really glad to see Python willing to take on these challenges even with 5 years of short-term pain, because it shows a commitment to keeping Python relevant for the issues that 21st-century programmers will face. An increasing number of software platforms will have to deal with non-English text; an increasing number will need to handle concurrent, event-based environments. Without these changes Python would basically cede these areas to languages like Go or Javascript that have their unicode story straight and are well-adapted to async programming.

philh 13y ago

> 3. In Python 2.c, warnings begun being issued when you tried to use the old way, explaining you needed to change or your code would stop working. > 4. In Python 2.d, it actually did stop working. I'm curious, what are some examples here? The `from __future__`s that I recall offhand are `print_function`, `division` (still need to be explicit) and the old `with_statement` (incompatible code broke as soon as this became default). The other old way of doing things that I can think of offhand is `catch Exception, e`, which has been replaced with `catch Exception as e`, but not removed.

sepeth 13y ago
there is `unicode_literals` too. For the full list, see http://docs.python.org/2/library/__future__.html http://docs.python.org/2/library/__future__.html And there is one forgotten in that list: from __future__ import braces
- philh 13y ago
  So it looks like in Aaron's list, 2.a == 2.c and 2.b == 2.d, and the process has been completed only three times, and two of those times were just adding keywords.

glynjackson 13y ago

quote from python.org: "New to Python or choosing between Python 2 and Python 3? Read Python 2 or Python 3." "Which version you ought to use is mostly dependent on what you want to get done." No other language I have ever worked with speaks like this. "New to Rudy? You could use version 2.1 or why not try version 1.0, it still works ok" Python.org treats version 2 and 3 as completely different things, newbies to the language like myself don't see it as a update to Python 2.x because thats not how it's sold to us.

jbeja 13y ago

I think the only people nthat hate python 3 is the one that have worked with it for several years, newbies wouldn't even care a little.

codecondo 13y ago

This aaron guy likes to talk a lot of shit so it seems