5 ms·

Porting to Python 3: An offer for you

29 points by llambda 15y ago

bdarnell 15y ago
A lot of the recent discussion around python3 has been about how 2to3 is slow and inconvenient, so this seems like a good time to plug my attempt to improve the process. https://github.com/bdarnell/auto2to3 https://github.com/bdarnell/auto2to3 uses an import hook to run 2to3 on changed files automatically, and cache the results on disk between runs so it doesn't have to run 2to3 over the entire project every time (and the caches are ordinary files so you can see exactly what 2to3 is producing). This is how I added Python 3 support to Tornado. If you intend to keep supporting Python 2.5, I definitely recommend using 2to3 instead of the contortions required to use a single codebase across all versions (if you're only supporting 2.6+ it gets much easier to do everything in one codebase without 2to3, although even then 2to3 can help keep the primary (2.x) codebase cleaner).
- saurik 15y ago
  The entire concept of 2to3, however, was stupid... what you need is 3to2. You want to be encouraging people to write new code, and support old versions as best as possible, not write old code and support new versions as best as possible. 2to3 should be used as a one-off migration tool, but it is seriously suggested to use it as part of people's builds.
  - bdarnell 15y ago
    In the long term I think we need both 2to3 and 3to2, but for now 2to3 is more important. As long as both my dependencies and the users of my library are mostly on 2.x, I want that to be my primary development platform. If the only way to support both platforms was to do most of my work in python 3 and then backport to 2.x, I'd have never gotten started because that's too much of a change to take on at once. 2to3 gives me a way to support 3.x now (even as a second class citizen), and then a path to flip things around and make 3.x the primary platform when I'm ready.
    - saurik 15y ago
      I just find the "for now" pretty hilarious: we are already seeing a shipping version of Python 3.2, with Python 3.x having been released for years, and we still need to encourage people to develop code for Python 2.x and use a build step to convert their code to Python 3.x. The effect of this attitude directly contributes to the behavior described in that recent article by the Jinja2 developer: where Python 3.x support is often considered by people to be just "look, I did it, and while it is a horrible hack and much slower than the 2.x version, it sort of works; now leave me alone", as opposed to helping people slowly embrace Python 3.x.
- zzzeek 15y ago
  I was just thinking of mocking up something like this...so why don't you write us up a blog post about this and get a little publicity ? If we could add on an "auto2to3" as easily as we build a virtualenv this has the potential to neutralize the "it's too hard to wait for 2to3" complaint.
phzbOx 15y ago
My 2 cents: stick with 2.7, it works well with all libraries and it's well supported everywhere. 3.0 provides nothing worth wasting million dollars on. The current mentality is "Avoid adding new feature to P3.0 to give time for people to port to it". What's happening? "Why would I switch to P3 if there's nothing worth switching for."
- regebro 15y ago
  That is incorrect, there is no such mentality and has never been any such mentality. Of course no new features is added to 3.0, because it's unsupported nowadays, Python 3 is up to 3.2, and 3.3 is coming soon. There was a moratorium for Python 3.2 where no more features was added to the language, to get other implementations than CPython a chance to catch up. But that was not a question of porting, and that did not include new features in general, just to the language. New features in the standard library and other implementation improvements was done in 3.2, and in 3.3 new language features are added again. So the claim of a mentality of "avoid adding new features" is nothing but FUD and complete hogwash.
pak 15y ago
The problem is, people have been told for so long not to care about Python 3 that it's hard to imagine what could make people actually start caring about it. Practically the only people that have even tried it are Python fanatics or compiler devs. I understand the desire to make fundamental language changes and why it's cleaner to make Python 3 backward incompatible. I don't agree that it was the smartest thing to prevent any kind of interoperability, though. The way people could begin using Python 3 is if you could still import Python 2 modules from Python 3 code. OK, there would be some tough nuts to crack to get the interpreter to switch contexts gracefully and translate objects from one side to the other. But at least people could start using Python 3 and then begin switching things over one at a time. To ask everyone to keep running 2to3 to maintain a second codebase that nobody uses is kind of silly and unrealistic, and causes this détente where too few people care enough to make a first move. Also, gripe here. Why the heck is def method(self, actual_first_arg, ...) still all over the place in Python 3? They had a golden opportunity to kick that bizarre holdover from when Python couldn't do OO [1], and they didn't [2]. The only solid argument Guido has against it is about decorators, and it really only applies to @classmethod and @staticmethod, which arises primarily from the stodgy way Python implements these OO features and which could have been nipped in the bud with their own language changes. [1] http://beust.com/weblog/2008/10/28/pythons-enigmatic-self/ http://beust.com/weblog/2008/10/28/pythons-enigmatic-self/ [2] http://neopythonic.blogspot.com/2008/10/why-explicit-self-has-to-stay.html http://neopythonic.blogspot.com/2008/10/why-explicit-self-ha...
- zzzeek 15y ago
  I love `self`. If it were just magically there, that would break with the entire rest of how the language works - everything is explicitly from something - even the "built in" functions are right there from __builtins__. What appeals to me about Python is the namespacing model is completely consistent throughout and explicit `self` is part of that.
  - pak 15y ago
    So, the namespacing is completely consistent, but the number of arguments between method definition and method calling is not? That breaks consistency both internally (because regular functions do have the same number of arguments) and with just about every other language I've used. Something tells me adding one little keyword that pokes into the namespace would not be the end of the world. It would, however, save me (and anyone else that has to juggle multiple languages) from several groaner bugs per day. Guido can repeat as many times as he likes that calling an instance method is morally equivalent to passing the instance as an additional argument. It may even be implemented that way. The reality is that nobody except compiler devs actually think like that--to everyone else it's an implementation detail. It also doesn't naturally explain why it would be shoved in as the first argument as opposed to appended as the last. And the fact that self is an idiom, and not a keyword, just feels wrong to me. If nobody would actually ever use any other word in sane code, there is no point in preserving the flexibility of doing so by defining the same word over and over again.

29 points by llambda 15y ago

A lot of the recent discussion around python3 has been about how 2to3 is slow and inconvenient, so this seems like a good time to plug my attempt to improve the process. https://github.com/bdarnell/auto2to3 https://github.com/bdarnell/auto2to3 uses an import hook to run 2to3 on changed files automatically, and cache the results on disk between runs so it doesn't have to run 2to3 over the entire project every time (and the caches are ordinary files so you can see exactly what 2to3 is producing). This is how I added Python 3 support to Tornado. If you intend to keep supporting Python 2.5, I definitely recommend using 2to3 instead of the contortions required to use a single codebase across all versions (if you're only supporting 2.6+ it gets much easier to do everything in one codebase without 2to3, although even then 2to3 can help keep the primary (2.x) codebase cleaner).

saurik 15y ago
The entire concept of 2to3, however, was stupid... what you need is 3to2. You want to be encouraging people to write new code, and support old versions as best as possible, not write old code and support new versions as best as possible. 2to3 should be used as a one-off migration tool, but it is seriously suggested to use it as part of people's builds.
- bdarnell 15y ago
  In the long term I think we need both 2to3 and 3to2, but for now 2to3 is more important. As long as both my dependencies and the users of my library are mostly on 2.x, I want that to be my primary development platform. If the only way to support both platforms was to do most of my work in python 3 and then backport to 2.x, I'd have never gotten started because that's too much of a change to take on at once. 2to3 gives me a way to support 3.x now (even as a second class citizen), and then a path to flip things around and make 3.x the primary platform when I'm ready.
  - saurik 15y ago
    I just find the "for now" pretty hilarious: we are already seeing a shipping version of Python 3.2, with Python 3.x having been released for years, and we still need to encourage people to develop code for Python 2.x and use a build step to convert their code to Python 3.x. The effect of this attitude directly contributes to the behavior described in that recent article by the Jinja2 developer: where Python 3.x support is often considered by people to be just "look, I did it, and while it is a horrible hack and much slower than the 2.x version, it sort of works; now leave me alone", as opposed to helping people slowly embrace Python 3.x.
zzzeek 15y ago
I was just thinking of mocking up something like this...so why don't you write us up a blog post about this and get a little publicity ? If we could add on an "auto2to3" as easily as we build a virtualenv this has the potential to neutralize the "it's too hard to wait for 2to3" complaint.

phzbOx 15y ago

My 2 cents: stick with 2.7, it works well with all libraries and it's well supported everywhere. 3.0 provides nothing worth wasting million dollars on. The current mentality is "Avoid adding new feature to P3.0 to give time for people to port to it". What's happening? "Why would I switch to P3 if there's nothing worth switching for."

regebro 15y ago
That is incorrect, there is no such mentality and has never been any such mentality. Of course no new features is added to 3.0, because it's unsupported nowadays, Python 3 is up to 3.2, and 3.3 is coming soon. There was a moratorium for Python 3.2 where no more features was added to the language, to get other implementations than CPython a chance to catch up. But that was not a question of porting, and that did not include new features in general, just to the language. New features in the standard library and other implementation improvements was done in 3.2, and in 3.3 new language features are added again. So the claim of a mentality of "avoid adding new features" is nothing but FUD and complete hogwash.

pak 15y ago

The problem is, people have been told for so long not to care about Python 3 that it's hard to imagine what could make people actually start caring about it. Practically the only people that have even tried it are Python fanatics or compiler devs. I understand the desire to make fundamental language changes and why it's cleaner to make Python 3 backward incompatible. I don't agree that it was the smartest thing to prevent any kind of interoperability, though. The way people could begin using Python 3 is if you could still import Python 2 modules from Python 3 code. OK, there would be some tough nuts to crack to get the interpreter to switch contexts gracefully and translate objects from one side to the other. But at least people could start using Python 3 and then begin switching things over one at a time. To ask everyone to keep running 2to3 to maintain a second codebase that nobody uses is kind of silly and unrealistic, and causes this détente where too few people care enough to make a first move. Also, gripe here. Why the heck is def method(self, actual_first_arg, ...) still all over the place in Python 3? They had a golden opportunity to kick that bizarre holdover from when Python couldn't do OO [1], and they didn't [2]. The only solid argument Guido has against it is about decorators, and it really only applies to @classmethod and @staticmethod, which arises primarily from the stodgy way Python implements these OO features and which could have been nipped in the bud with their own language changes. [1] http://beust.com/weblog/2008/10/28/pythons-enigmatic-self/ http://beust.com/weblog/2008/10/28/pythons-enigmatic-self/ [2] http://neopythonic.blogspot.com/2008/10/why-explicit-self-has-to-stay.html http://neopythonic.blogspot.com/2008/10/why-explicit-self-ha...

zzzeek 15y ago
I love `self`. If it were just magically there, that would break with the entire rest of how the language works - everything is explicitly from something - even the "built in" functions are right there from __builtins__. What appeals to me about Python is the namespacing model is completely consistent throughout and explicit `self` is part of that.
- pak 15y ago
  So, the namespacing is completely consistent, but the number of arguments between method definition and method calling is not? That breaks consistency both internally (because regular functions do have the same number of arguments) and with just about every other language I've used. Something tells me adding one little keyword that pokes into the namespace would not be the end of the world. It would, however, save me (and anyone else that has to juggle multiple languages) from several groaner bugs per day. Guido can repeat as many times as he likes that calling an instance method is morally equivalent to passing the instance as an additional argument. It may even be implemented that way. The reality is that nobody except compiler devs actually think like that--to everyone else it's an implementation detail. It also doesn't naturally explain why it would be shoved in as the first argument as opposed to appended as the last. And the fact that self is an idiom, and not a keyword, just feels wrong to me. If nobody would actually ever use any other word in sane code, there is no point in preserving the flexibility of doing so by defining the same word over and over again.