5 ms·

I've been doing this for ten years. Why wouldn't you use 2to3? In my experience, 2to3 is all you need 95% of the time. It is really that good. The only time

I've been doing this for ten years. Why wouldn't you use 2to3? In my experience, 2to3 is all you need 95% of the time. It is really that good. The only times it doesn't work is with python that is bad to an unusual degree. For example, if you make a variable named "list", overriding the built-in list type. 2to3 freaks right out. The other type of "bad" python is stuff that abuses the "open kimono" nature of the language and heavily monkeypatches python internals. Or things that depend on version specific binary data, like pickle files.

deleted 9y ago
[deleted]
jwilk 9y ago
> 2to3 is all you need 95% of the time This is absolutely not the case in my experience. 2to3 tends to produce something that appears to work at first glance, but it actually doesn't.
- aaossa 9y ago
  You have any examples? I was planning to use 2to3 in a project, can you explain in which case it doesn't work? Thanks!
  - masklinn 9y ago
    First, I'd recommend python-future rather than 2to3, it has additional fixers and replaces some with better versions. Second, those only do the fairly straightforward changes, fixers are simple AST transformations they don't do complex type analysis or anything, you should use them as a starting point but don't expect the work to be over by then: the 2->3 transitions has a number of API and semantics changes which these tools can't handle, text model changes are the biggest one[0] but they're not the only one by far. If the project is non-trivial, having a good test coverage is absolutely crucial. Basically, 2to3 handles the first 90% of the work which are mostly drudgery and syntactic fixes, you're still on the hook for the second 90% which is more subtle. Source: finalising the conversion of a ~200k SLOC codebase to cross-version compatibility. [0] not just the strict separation of text and bytes, but APIs being set to one or the other so you have places where you want to ensure bytes, others where you want to ensure text and yet others where you need "native strings", some APIs (csv) are also incompatibly altered.
  - tedunangst 9y ago
    Lots of people mention strings, but for a concreteish example: you have some code which reads bytes from the network, say "cookie", and uses that as a key into a dict. Later you try to access the stored value with headers["cookie"]. This works in python 2. It silently fails in python 3 because b"cookie" and "cookie" are different keys. Not really related to whether the original code supported Unicode or not.
- jxramos 9y ago
  2to3 gets all the good low hanging fruit syntatic fixes I've found, but it really comes down to the str/bytes divergence and how mucked up the mixing was in the py2 code. His comment here is the clinch... "pdbparse is among the worst offendants I’ve seen since, as any parsing library, it will happily handle and transform bytes-like data as string."
- acdha 9y ago
  This has not been accurate at all in my experience. It really came down to whether the code had supported non-ASCII text before — if there's a huge amount of bytes/str conflation, it's a mess but if it followed good Python 2 practice it's been very close to convert and run.
- svara 9y ago
  I agree, although different people's experience may differ simply because they were or were not using certain language features often. What bites me all the time is the division operator. The 2to3 code will run, but produce incorrect results. String / byte handling is an obvious other one.
- _paulc 9y ago
  This lines up with my experience. Strangely I had someone submit a bunch of slightly passive-aggressive "Py3 support" pull requests to a couple of my repos which were basically the result of running 2to3 on the code. They hadn't even bothered to run the unit tests (which obviously failed spectacularly). It did however prompt me to get Py3 support working properly which was actually a good thing.
  - iak8god 9y ago
    > Strangely I had someone submit a bunch of slightly passive-aggressive "Py3 support" pull requests Is there a more polite way to go about this? I submitted some "Py3 support"-style pull requests to a repo out of the blue a while back. In that case it was just a few minor changes to some fairly small scripts and I did those by hand, preserving Py2 compatibility. But I did wonder at the time whether it wasn't a bit jarring for the owner to see a stranger pop up with a bunch of pull requests like that.
    - deleted 9y ago
      [deleted]
    - walrus 9y ago
      > Is there a more polite way to go about this? This is the usual process for submitting a large change to a project: 1. Check the issue tracker or mailing list for an existing discussion about Python 3 support. If there is one, read the discussion. If there isn't one, start the discussion. Mention that you're willing to make the changes if there is interest. Remember that supporting both Python 2 and Python 3 places a burden on all future contributors. 2. Keep the changes minimal and easy to review. 3. Review the changes yourself before sending a patch/pull request. 4. Run the test suite. 5. Test your changes in all versions of Python the software supports. Forgetting #1 is usually the cause of tension when people submit large patches. The person who submits a patch wants to see their patch included. The authors of the project want to make sure it aligns with the goals of their project and doesn't break their software.
alexchamberlain 9y ago
I thought 2to3 was designed for converting apps from 2 to 3, rather than making them polyglot? I think six could have simplified things here, however.
- keenerd 9y ago
  You can't make something truly polyglot across all of 2.X and all of 3.X. However much of py3 syntax is supported by py2. As far as I know, 2to3 doesn't produce py3-only syntax. I diff the original and the converted code, merging the non-compatible parts into a version check.
  - masklinn 9y ago
    > You can't make something truly polyglot across all of 2.X and all of 3.X. There's little reason to though, generally you want 2.7 and >=3.x (the simplest is 3.5, 3.4 is fairly easy, 3.3 is possible but starts being more annoying) > As far as I know, 2to3 doesn't produce py3-only syntax. It does: print function, metaclass kwarg. It will also generate syntactically valid but P2-incompatible code (with_traceback calls) as well as possibly semantically break P2 code (rename __nonzero__ to __bool__, basestring and unicode to str, stdlib changes, raw_input to input, …)
    - keenerd 9y ago
      Print() is valid in Python2. It will print a tuple under some conditions, but that is still syntactically valid and a graceful degradation. (Print is for humans to read. As long as a human can understand it, good enough. Going for full identical behavior costs extra :-) Valid-but-incompatible is fine for polyglot stuff. Merge the files and put in a version test around those parts.
      - masklinn 9y ago
        > Print() is valid in Python2. print("a", end=" ") is not, neither is class Foo(metaclass=Bar)
- deleted 9y ago
trextrex 9y ago
I've found that python future [1] does a pretty remarkable job of converting code to be both python 2 and python 3 compatible. [1] http://python-future.org http://python-future.org

deleted 9y ago

[deleted]

jwilk 9y ago

> 2to3 is all you need 95% of the time This is absolutely not the case in my experience. 2to3 tends to produce something that appears to work at first glance, but it actually doesn't.

aaossa 9y ago
You have any examples? I was planning to use 2to3 in a project, can you explain in which case it doesn't work? Thanks!
- masklinn 9y ago
  First, I'd recommend python-future rather than 2to3, it has additional fixers and replaces some with better versions. Second, those only do the fairly straightforward changes, fixers are simple AST transformations they don't do complex type analysis or anything, you should use them as a starting point but don't expect the work to be over by then: the 2->3 transitions has a number of API and semantics changes which these tools can't handle, text model changes are the biggest one[0] but they're not the only one by far. If the project is non-trivial, having a good test coverage is absolutely crucial. Basically, 2to3 handles the first 90% of the work which are mostly drudgery and syntactic fixes, you're still on the hook for the second 90% which is more subtle. Source: finalising the conversion of a ~200k SLOC codebase to cross-version compatibility. [0] not just the strict separation of text and bytes, but APIs being set to one or the other so you have places where you want to ensure bytes, others where you want to ensure text and yet others where you need "native strings", some APIs (csv) are also incompatibly altered.
- tedunangst 9y ago
  Lots of people mention strings, but for a concreteish example: you have some code which reads bytes from the network, say "cookie", and uses that as a key into a dict. Later you try to access the stored value with headers["cookie"]. This works in python 2. It silently fails in python 3 because b"cookie" and "cookie" are different keys. Not really related to whether the original code supported Unicode or not.
jxramos 9y ago
2to3 gets all the good low hanging fruit syntatic fixes I've found, but it really comes down to the str/bytes divergence and how mucked up the mixing was in the py2 code. His comment here is the clinch... "pdbparse is among the worst offendants I’ve seen since, as any parsing library, it will happily handle and transform bytes-like data as string."
acdha 9y ago
This has not been accurate at all in my experience. It really came down to whether the code had supported non-ASCII text before — if there's a huge amount of bytes/str conflation, it's a mess but if it followed good Python 2 practice it's been very close to convert and run.
svara 9y ago
I agree, although different people's experience may differ simply because they were or were not using certain language features often. What bites me all the time is the division operator. The 2to3 code will run, but produce incorrect results. String / byte handling is an obvious other one.
_paulc 9y ago
This lines up with my experience. Strangely I had someone submit a bunch of slightly passive-aggressive "Py3 support" pull requests to a couple of my repos which were basically the result of running 2to3 on the code. They hadn't even bothered to run the unit tests (which obviously failed spectacularly). It did however prompt me to get Py3 support working properly which was actually a good thing.
- iak8god 9y ago
  > Strangely I had someone submit a bunch of slightly passive-aggressive "Py3 support" pull requests Is there a more polite way to go about this? I submitted some "Py3 support"-style pull requests to a repo out of the blue a while back. In that case it was just a few minor changes to some fairly small scripts and I did those by hand, preserving Py2 compatibility. But I did wonder at the time whether it wasn't a bit jarring for the owner to see a stranger pop up with a bunch of pull requests like that.
  - deleted 9y ago
    [deleted]
  - walrus 9y ago
    > Is there a more polite way to go about this? This is the usual process for submitting a large change to a project: 1. Check the issue tracker or mailing list for an existing discussion about Python 3 support. If there is one, read the discussion. If there isn't one, start the discussion. Mention that you're willing to make the changes if there is interest. Remember that supporting both Python 2 and Python 3 places a burden on all future contributors. 2. Keep the changes minimal and easy to review. 3. Review the changes yourself before sending a patch/pull request. 4. Run the test suite. 5. Test your changes in all versions of Python the software supports. Forgetting #1 is usually the cause of tension when people submit large patches. The person who submits a patch wants to see their patch included. The authors of the project want to make sure it aligns with the goals of their project and doesn't break their software.

alexchamberlain 9y ago

I thought 2to3 was designed for converting apps from 2 to 3, rather than making them polyglot? I think six could have simplified things here, however.

keenerd 9y ago
You can't make something truly polyglot across all of 2.X and all of 3.X. However much of py3 syntax is supported by py2. As far as I know, 2to3 doesn't produce py3-only syntax. I diff the original and the converted code, merging the non-compatible parts into a version check.
- masklinn 9y ago
  > You can't make something truly polyglot across all of 2.X and all of 3.X. There's little reason to though, generally you want 2.7 and >=3.x (the simplest is 3.5, 3.4 is fairly easy, 3.3 is possible but starts being more annoying) > As far as I know, 2to3 doesn't produce py3-only syntax. It does: print function, metaclass kwarg. It will also generate syntactically valid but P2-incompatible code (with_traceback calls) as well as possibly semantically break P2 code (rename __nonzero__ to __bool__, basestring and unicode to str, stdlib changes, raw_input to input, …)
  - keenerd 9y ago
    Print() is valid in Python2. It will print a tuple under some conditions, but that is still syntactically valid and a graceful degradation. (Print is for humans to read. As long as a human can understand it, good enough. Going for full identical behavior costs extra :-) Valid-but-incompatible is fine for polyglot stuff. Merge the files and put in a version test around those parts.
    - masklinn 9y ago
      > Print() is valid in Python2. print("a", end=" ") is not, neither is class Foo(metaclass=Bar)
deleted 9y ago

trextrex 9y ago

I've found that python future [1] does a pretty remarkable job of converting code to be both python 2 and python 3 compatible. [1] http://python-future.org http://python-future.org