6 ms·

The whole Python versioning fiasco is simply baffling to me as an outsider. Why not mark Python 3 modules with a tag, or different file extension, or anything,

The whole Python versioning fiasco is simply baffling to me as an outsider. Why not mark Python 3 modules with a tag, or different file extension, or anything, and require the Python 3 interpreter to be able to interpret Python 2 code as well as Python 3 code? This is e.g. how the (roughly analogous) split between C and C++ is handled, and it works fine for the most part. No bizarre polyglot code games. Edit: Just to be clear, C++ is not a superset of C: http://stackoverflow.com/a/1201840/270610 http://stackoverflow.com/a/1201840/270610

orf 11y ago
> require the Python 3 interpreter to be able to interpret Python 2 code as well as Python 3 code Because that would make the interpreter a whole lot more complicated. A lot of technical debt was cleared with the 2-3 transition, adding it back in defeats the point. C/C++ is different, C++ is a superset of C. That makes things a lot easier. Edit: Ok, C++ is not a superset, common misconception.
- m3koval 11y ago
  C++ is not technically a superset of C. Here are a few examples that are valid in C, but not C++: http://stackoverflow.com/a/1201840/111426 http://stackoverflow.com/a/1201840/111426 However, it's close enough that I generally agree with your point.
- hellofunk 11y ago
  >C/C++ is different, C++ is a superset of C. That makes things a lot easier. This is entirely incorrect, but I'll give you a break since it is a common misconception. C and C++ are truly two different languages. C is not 100% compatible with C++.
- thaumasiotes 11y ago
  > C++ is a superset of C No it isn't.
takeda 11y ago
Because you actually can write code that is compatible with both Python 2 and 3. Also major issues people have with migration is unicode. In respect of unicode, majority of the code is simply broken and fails to work on Python 3 because Python 3 is more strict about it. In Python 2 you store text and binary data as bytes (no distinction, there is unicode type but it's more hassle to use it and frankly almost no one does use it) while in Python 3 text and binary data are two distinct types. So if you have code in Python 2 and you did not distinguish what is text and what is binary now you have to go through your entire code and identify all those parts, this is especially hard since Python is dynamically typed language and you don't have type checker to help you with that. This is also why it is much easier to write Python 3 code and make it run on Python 2 as well than the other way around. As for having interpreter compatibility layer. First python does not care about extension you use, and unlike some languages it actually allows you to install multiple versions at the same time without conflicts (checked 2.4 all the way to 3.5). You can easily control which version you want to use by shabang, for example: #! /usr/bin/env python2 #! /usr/bin/env python3 #! /usr/bin/env python3.5
- colanderman 11y ago
  You can also write code that is compatible with both C and Perl. Doesn't make it a good idea. Shebang doesn't work on a per-module basis, which is what I am advocating. If there were a way to specify versions at that granularity, there would be no need for 2/3 polyglot tricks.
  - takeda 11y ago
    That's a strawman argument. Majority of python libraries are written that way. I guess I'm incredibly lucky or you're incredibly unlucky, because I did not run into a library that would only run on Python 2. Ok, I did run into some, but those were no longer maintained and frankly I would not use them even on Python 2.
    - berntb 11y ago
      So the problems with upgrading from Python 2 to 3 because of library support are over? Including linked C code? Or is that just your lucky experience?
rogerbinns 11y ago
Are you expecting the same running interpreter to have both python 2 and 3 modules at the same time, with calls between the modules working? You can easily keep things separate - python 2 executable is named python while python 3 is python3. https://www.python.org/dev/peps/pep-0397/ https://www.python.org/dev/peps/pep-0397/ Sadly concurrent running in the same interpreter won't work because Python 3 has different types for strings (unicode only) and a bytes type, while Python 2 has a str type (pretty much like bytes but can also be treated as a string) and a separate unicode type. Python 2 had code that tried to do the right thing automagically with str, such as automatic promotion to unicode where it looks like you really meant a string and that is what is needed. Automatic promotion can fail, be unexpected, is hard to test and numerous other issues. That is why Python 3 was needed and why it needed to be incompatible. To have Python 2 and 3 code interoperate in the same interpreter would require another layer of automatic promotion/demotion and heuristics when data goes between the two. C++ was explicitly defined to be a superset of C so it doesn't have these kind of issues. The analogous situation would be if C didn't distinguish between long and pointer types, and C++ did. You'd need rules for how to convert between the two worlds, trying to combine or disambiguate the types depending on which way types are going. It would be a mess.
- colanderman 11y ago
  Yes, that is what I expect. The various flavors of C and C++ get along just fine that way. C++ is not a superset of C: http://stackoverflow.com/a/1201840/270610 http://stackoverflow.com/a/1201840/270610 C and C++ have a similar issue with strings. C code uses NUL-terminated char arrays, while C++ generally uses std::string. If you want to interface the two, you need glue code (written in C++). Surely Python 3 can work the same way: to share strings between a Python 2 and Python 3 module, you need to convert them using glue code written in Python 3. Sure it's messy, but writing polyglot code is messier.
  - deleted 11y ago
    [deleted]
  - hyperpape 11y ago
    I wonder if there's even a reasonable way to do this in Python. In C, strings are data that you're explicitly passing around, but in Python, you are constantly calling __getattribute__ under the hood just to execute your code. I don't know that it would be impossible to make that all work, but I wouldn't assume that it is.
the_mitsuhiko 11y ago
> and require the Python 3 interpreter to be able to interpret Python 2 code as well as Python 3 code? The interpreter is too messy for that and the transition to Python 3 was not considered well enough. There is no hope on that front. Even if you solve everything else, the C ABI will still be broken.
sandGorgon 11y ago
I think this is fundamentally because of the Python philosophy around one-true-way to do things (something that's open to interpretation given the number of string functions that are around). IMHO, the Python 3 vs Python 2 split is a phlisophical rift rather than a technical one. There is no real technical reasons why print and print() cannot coexist. In all reality, Python 2 and Python 3 are completely differently languages at this point. Communities/companies/projects that have adopted Python 2 do not care about anything that Python 3 gives them. Take for instance Flask - to maintain compatibility with the WSGI spec they have to maintain Python 2 compatibility indefinitely. Less than 30% of all software is compatible with Py3 and any new software releases are built on top of Python 2 - yes I'm looking at Tensorflow. I'm very doubtful that Python Foundation is going to drop support by 2020. Quite simply - a highly funded Python company (like anaconda) can create a fork. The only way this situation can be fixed is Python 4 with co-existence of Python 3 and Python 2 functions.
- colanderman 11y ago
  As an aside, this: > the Python philosophy around one-true-way to do things I've always thought to be a total joke. There's been like 5 different modules each for string interpolation, subprocess management, and argument parsing. Heck, I thought I was on top of which was in vogue and apparently just two weeks ago there's a new "one true way" to do string interpolation.
  - chishaku 11y ago
    I came across this recently: > Barry Warsaw, one of the core Python developers, once said that it frustrated him that "The Zen of Python" (PEP 20) is used as a style guide for Python code, since it was originally written as a poem about Python's internal design. [0] I did a quick search for a little more background but didn't find anything. I'd be curious to hear more. [0]: https://github.com/amontalenti/elements-of-python-style#a-little-zen-for-your-code-style https://github.com/amontalenti/elements-of-python-style#a-li...
  - such_a_casual 11y ago
    I think this needs to be the responsibility of things like PEP rather than the actual language. Otherwise you break old code every time you find a better way.
unfamiliar 11y ago
I've always wondered this. Why not have .py and .py3 extensions, and require the python 3 interpreter to fall back to Python 2? That would also solve the problem of "which version of Python is this script I found written in?"
eevee 11y ago
What do you do when a value passes between major versions which have different implementations of its type? What do you do with the `int` type, which was actually two separate types in Python 2? What do you do when the binary representation of strings changes (as it did in the 3 series) and you want to pass that into Python 2 land? What do you do about extension modules written in C, which many popular Python libraries use to speed up hot code? Do you include copies of both standard libraries, meaning you now have duplicate modules and may get one or the other depending on the tag in your file? Which version of Python 2 do you target? What happens when there's a bug in one Python 2 implementation but not the other? How do you handle having two C APIs at the same time? You mention C++ as though it worked out great, but C++ is still full of sharp edges and features that interact weirdly with its C legacy, and it has never completely replaced C (especially for libraries). And that's despite having a few advantages that made it possible in the first place, like having a completely different stdlib built on top of the C stdlib, and producing machine code rather than compiling for a VM on the fly.
dilap 11y ago
yeah, the python folks lost their mind around the 2/3 split. source: ex-python user.

orf 11y ago

> require the Python 3 interpreter to be able to interpret Python 2 code as well as Python 3 code Because that would make the interpreter a whole lot more complicated. A lot of technical debt was cleared with the 2-3 transition, adding it back in defeats the point. C/C++ is different, C++ is a superset of C. That makes things a lot easier. Edit: Ok, C++ is not a superset, common misconception.

m3koval 11y ago
C++ is not technically a superset of C. Here are a few examples that are valid in C, but not C++: http://stackoverflow.com/a/1201840/111426 http://stackoverflow.com/a/1201840/111426 However, it's close enough that I generally agree with your point.
hellofunk 11y ago
>C/C++ is different, C++ is a superset of C. That makes things a lot easier. This is entirely incorrect, but I'll give you a break since it is a common misconception. C and C++ are truly two different languages. C is not 100% compatible with C++.
thaumasiotes 11y ago
> C++ is a superset of C No it isn't.

takeda 11y ago

Because you actually can write code that is compatible with both Python 2 and 3. Also major issues people have with migration is unicode. In respect of unicode, majority of the code is simply broken and fails to work on Python 3 because Python 3 is more strict about it. In Python 2 you store text and binary data as bytes (no distinction, there is unicode type but it's more hassle to use it and frankly almost no one does use it) while in Python 3 text and binary data are two distinct types. So if you have code in Python 2 and you did not distinguish what is text and what is binary now you have to go through your entire code and identify all those parts, this is especially hard since Python is dynamically typed language and you don't have type checker to help you with that. This is also why it is much easier to write Python 3 code and make it run on Python 2 as well than the other way around. As for having interpreter compatibility layer. First python does not care about extension you use, and unlike some languages it actually allows you to install multiple versions at the same time without conflicts (checked 2.4 all the way to 3.5). You can easily control which version you want to use by shabang, for example: #! /usr/bin/env python2 #! /usr/bin/env python3 #! /usr/bin/env python3.5

colanderman 11y ago
You can also write code that is compatible with both C and Perl. Doesn't make it a good idea. Shebang doesn't work on a per-module basis, which is what I am advocating. If there were a way to specify versions at that granularity, there would be no need for 2/3 polyglot tricks.
- takeda 11y ago
  That's a strawman argument. Majority of python libraries are written that way. I guess I'm incredibly lucky or you're incredibly unlucky, because I did not run into a library that would only run on Python 2. Ok, I did run into some, but those were no longer maintained and frankly I would not use them even on Python 2.
  - berntb 11y ago
    So the problems with upgrading from Python 2 to 3 because of library support are over? Including linked C code? Or is that just your lucky experience?

rogerbinns 11y ago

Are you expecting the same running interpreter to have both python 2 and 3 modules at the same time, with calls between the modules working? You can easily keep things separate - python 2 executable is named python while python 3 is python3. https://www.python.org/dev/peps/pep-0397/ https://www.python.org/dev/peps/pep-0397/ Sadly concurrent running in the same interpreter won't work because Python 3 has different types for strings (unicode only) and a bytes type, while Python 2 has a str type (pretty much like bytes but can also be treated as a string) and a separate unicode type. Python 2 had code that tried to do the right thing automagically with str, such as automatic promotion to unicode where it looks like you really meant a string and that is what is needed. Automatic promotion can fail, be unexpected, is hard to test and numerous other issues. That is why Python 3 was needed and why it needed to be incompatible. To have Python 2 and 3 code interoperate in the same interpreter would require another layer of automatic promotion/demotion and heuristics when data goes between the two. C++ was explicitly defined to be a superset of C so it doesn't have these kind of issues. The analogous situation would be if C didn't distinguish between long and pointer types, and C++ did. You'd need rules for how to convert between the two worlds, trying to combine or disambiguate the types depending on which way types are going. It would be a mess.

colanderman 11y ago
Yes, that is what I expect. The various flavors of C and C++ get along just fine that way. C++ is not a superset of C: http://stackoverflow.com/a/1201840/270610 http://stackoverflow.com/a/1201840/270610 C and C++ have a similar issue with strings. C code uses NUL-terminated char arrays, while C++ generally uses std::string. If you want to interface the two, you need glue code (written in C++). Surely Python 3 can work the same way: to share strings between a Python 2 and Python 3 module, you need to convert them using glue code written in Python 3. Sure it's messy, but writing polyglot code is messier.
- deleted 11y ago
  [deleted]
- hyperpape 11y ago
  I wonder if there's even a reasonable way to do this in Python. In C, strings are data that you're explicitly passing around, but in Python, you are constantly calling __getattribute__ under the hood just to execute your code. I don't know that it would be impossible to make that all work, but I wouldn't assume that it is.

the_mitsuhiko 11y ago

> and require the Python 3 interpreter to be able to interpret Python 2 code as well as Python 3 code? The interpreter is too messy for that and the transition to Python 3 was not considered well enough. There is no hope on that front. Even if you solve everything else, the C ABI will still be broken.

sandGorgon 11y ago

I think this is fundamentally because of the Python philosophy around one-true-way to do things (something that's open to interpretation given the number of string functions that are around). IMHO, the Python 3 vs Python 2 split is a phlisophical rift rather than a technical one. There is no real technical reasons why print and print() cannot coexist. In all reality, Python 2 and Python 3 are completely differently languages at this point. Communities/companies/projects that have adopted Python 2 do not care about anything that Python 3 gives them. Take for instance Flask - to maintain compatibility with the WSGI spec they have to maintain Python 2 compatibility indefinitely. Less than 30% of all software is compatible with Py3 and any new software releases are built on top of Python 2 - yes I'm looking at Tensorflow. I'm very doubtful that Python Foundation is going to drop support by 2020. Quite simply - a highly funded Python company (like anaconda) can create a fork. The only way this situation can be fixed is Python 4 with co-existence of Python 3 and Python 2 functions.

colanderman 11y ago
As an aside, this: > the Python philosophy around one-true-way to do things I've always thought to be a total joke. There's been like 5 different modules each for string interpolation, subprocess management, and argument parsing. Heck, I thought I was on top of which was in vogue and apparently just two weeks ago there's a new "one true way" to do string interpolation.
- chishaku 11y ago
  I came across this recently: > Barry Warsaw, one of the core Python developers, once said that it frustrated him that "The Zen of Python" (PEP 20) is used as a style guide for Python code, since it was originally written as a poem about Python's internal design. [0] I did a quick search for a little more background but didn't find anything. I'd be curious to hear more. [0]: https://github.com/amontalenti/elements-of-python-style#a-little-zen-for-your-code-style https://github.com/amontalenti/elements-of-python-style#a-li...
- such_a_casual 11y ago
  I think this needs to be the responsibility of things like PEP rather than the actual language. Otherwise you break old code every time you find a better way.

unfamiliar 11y ago

I've always wondered this. Why not have .py and .py3 extensions, and require the python 3 interpreter to fall back to Python 2? That would also solve the problem of "which version of Python is this script I found written in?"

eevee 11y ago

What do you do when a value passes between major versions which have different implementations of its type? What do you do with the `int` type, which was actually two separate types in Python 2? What do you do when the binary representation of strings changes (as it did in the 3 series) and you want to pass that into Python 2 land? What do you do about extension modules written in C, which many popular Python libraries use to speed up hot code? Do you include copies of both standard libraries, meaning you now have duplicate modules and may get one or the other depending on the tag in your file? Which version of Python 2 do you target? What happens when there's a bug in one Python 2 implementation but not the other? How do you handle having two C APIs at the same time? You mention C++ as though it worked out great, but C++ is still full of sharp edges and features that interact weirdly with its C legacy, and it has never completely replaced C (especially for libraries). And that's despite having a few advantages that made it possible in the first place, like having a completely different stdlib built on top of the C stdlib, and producing machine code rather than compiling for a VM on the fly.

dilap 11y ago

yeah, the python folks lost their mind around the 2/3 split. source: ex-python user.