10 ms·

I've seen this story happen twice before. 20 years ago there was this great MP3 player, WinAmp 2. And then they released WinAmp 3, which broke compatibility wi

by 21 10y ago

I've seen this story happen twice before. 20 years ago there was this great MP3 player, WinAmp 2. And then they released WinAmp 3, which broke compatibility with skins and plugins and which was slow. People didn't upgrade. Finally, they came to their senses and they released WinAmp 5, marketed as 2+3, which was faster and brought back compatibility with older stuff. In the retail trading Forex world, there is this great trading app called MetaTrader 4. Many years ago they released MetaTrader 5, which broke backward compatibility and removed some popular features. People didn't upgrade. Today, they are finally bringing back the removed features and making MetaTrader 5 able to run MetaTrader 4 code. Me, I wait for Python 5 (=2+3) which will be able to import Python 2 modules, so that you can gradually convert your code to the newer version. I have tens of thousands of lines of Python 2 code in a big system. I can't just take 2 months off to move all of it to Python 3. Moving it in pieces is also not really possible, since there are many inter-dependencies. Uglyfying my code with the "six" module it's also not a solution, since when I'll move, I won't care about Python 2 anymore. So basically I'm just waiting until a consensus emerges. It should be noted that the quantity of Python 2 code keeps on growing, I wrote most of my code while Python 3 existed. If Python 3 allowed an easy path forward, we wouldn't be in this situation.

takeda 10y ago
The biggest obstacle people seem to complain about (besides print, which is trivial IMO) is unicode. This is generally also the most painful part of making python 2 code work on python 3. The issue with it is that from unicode perspective your python 2 code is inherently broken. There's no automated tool that can fix it, and there's no way python 5 will be able to run python 2 code correctly unless unicode support would be removed. You have to manually tell python which string is made of characters and which one was made out of bytes.
ash 10y ago
> Me, I wait for Python 5 (=2+3) which will be able to import Python 2 modules, so that you can gradually convert your code to the newer version. Thank you for saying this! For those interested in this direction, there was a discussion about it on HN a couple of months ago: https://news.ycombinator.com/item?id=10823406 https://news.ycombinator.com/item?id=10823406 Proof-of-concept examples embedding Python interpreters within themselves: https://gist.github.com/dutc/eba9b2f7980f400f6287 https://gist.github.com/dutc/eba9b2f7980f400f6287 https://gist.github.com/dutc/2866d969d5e9209d501a https://gist.github.com/dutc/2866d969d5e9209d501a
mixmastamyk 10y ago
> Uglyfying my code with the "six" module it's also not a solution, since when I'll move, I won't care about Python 2 anymore. I've upgraded a lot of my code with just future imports and it is just a tiny step from there, unless you're doing a lot of character encoding. Leave those projects behind, but moving most other projects is easier than I expected.
klodolph 10y ago
All of the reasonable "2+3" features in Python have already landed, like u'' strings in Python 3.3. The really big issues, like the difference between strings and bytes objects, can't really be emulated away. This isn't like WinAmp or MetaTrader where you can just reimplement the old interface in the new application. Code that mixes strings and bytestrings freely is simply not going to interface well with Python 3 code. There are few paths for upgrading. You haven't mentioned 2to3, but I will. I've migraded projects of the same size you mention (10s of kloc) with 2to3 and the process took days, not months. About three days with one developer, actually. There are a number of projects which also have code bases which are compatible with both Python 2 and 3. You say that once you move, you won't care about Python 2 any more, but I that's not a good reason to reject six outright. In my experience, the code isn't really "uglified" but there are just a few minor cases here and there you need to think about. Anyway. Small-ish projects like the one you are talking about are usually not as hard to migrate as you might think. There are some exceptions, of course.
- xapata 10y ago
  > There are some exceptions, of course. Those exceptions are often bugs in the logic of handling text. Python 2 allows you to sweep that under the rug.
- 21 10y ago
  I agree that the bytes/string problem is a tough one. But the dev team created it and dropped it into the laps of it's users. The Windows API had the same problem, it was ASCII and then they added Unicode. But instead of just forcing your app to exclusively use a single one of them (sort of like use Py2 OR Py3), they allowed you to mix and match your code, and call either the old or the new version of the API. And gradually, people stopped using the older version and started using the Unicode one. Today, new Windows APIs are strictly Unicode. Of course, the solution was ugly from their side, having 2 duplicate APIs for the same thing, but it didn't brought pain to their users and allowed them to proceed at leisure.
  - kstrauser 10y ago
    So, it's like Python. You can still mix and match; you just have to explicitly convert between them at the appropriate points. The main breaking change is that you can't read a stream of bytes from a byte-oriented resource (say, a network connection that by definition can only send a stream of octets) and magically treat it like text elsewhere. While that's a tempting thing to want to do, it's been an unending fountain of bugs over the years. For example, in Python 2.7: >>> a = 'this is a test ಠ_ಠ' >>> b = bytes(a) >>> b 'this is a test \xe0\xb2\xa0_\xe0\xb2\xa0' >>> type(b) <type 'str'> In Python 3.5: >>> a = 'this is a test ಠ_ಠ' >>> print(a) this is a test ಠ_ಠ >>> b = bytes(a) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: string argument without an encoding >>> b = bytes(a.encode('utf-8')) >>> b b'this is a test \xe0\xb2\xa0_\xe0\xb2\xa0' >>> type(b) <class 'bytes'> In 2.7, you encode a series of Unicode values back to... a string? This is the source of all kinds of fun as functions expecting a string as input don't have a way to tell whether it's already been encoded or if they need to do so. In 3.5, there's no ambiguity. Functions that write to files or network sockets can either receive strings and know that they have to encode the strings locally, or bytes and know that they don't have to. Right up near the top of Zen we get: >>> import this The Zen of Python, by Tim Peters Beautiful is better than ugly. Explicit is better than implicit. The Python 2-style implicit conversions were awfully convenient 15 years ago when most things were ASCII. They're absolutely painful now that we've moved past the expectation that strings are arrays of single bytes.
syllogism 10y ago
Cython can probably compile your existing codebase such that it's compatible with both 2 and 3. Then you can write your new code in Python 3, while importing the old code as a C extension. You can also do this piece-by-piece, if for some reason your code doesn't compile with Cython right away. The compilation process also makes it much easier to identify bytes/unicode bugs, because you can choose to declare types.
- dragonne 10y ago
  That sounds like an interesting approach. Have you tried it with a real-world project? I'm curious if the approach has been proven.
luch 10y ago
I've used this module for upgrading a python2-only lib : https://pypi.python.org/pypi/future https://pypi.python.org/pypi/future . Basically it allow you to write Python 3 code that will run on Python 2 interpreter (bar from using advanced features like coroutine or matrix mult operator). Once you decide not to support Python2 anymore, just drop the imports. They also have a Py2/3 compat cheet sheet : http://python-future.org/compatible_idioms.pdf http://python-future.org/compatible_idioms.pdf
- daveguy 10y ago
  It is pretty obvious there is a problem when your quick and easy cheat sheet is a 33 page tomb. If only Guido and company would acknowledge the problem and fix it instead of putting their fingers in their ears pretending Python 2 will cease to exist in 2020.
  - wnoise 10y ago
    They acknowledge the problem, and their solution is to stop actively aiding and abetting the people who are continuing to make it worse by writing python 2 code.
    - nomel 10y ago
      Wow, what a simplification. What about us who have and want to use massive libraries of existing Python 2 code? "Just upgrade your libraries!" is not realistic for everyone, especially those who have to justify a mostly idealogical conversion to a similar language, and spend real time and money, outside of actually developing the end product, to do it. Four more years of official support is a long time.
      - takeda 10y ago
        Someone suggested Cython to compile python 2 code as modules and import to python 3. Did not try it, but that sounds plausible.
ericfrederich 10y ago
It should be noted that the quantity of Python 2 code keeps on growing According to this article Python2 support is shrinking. https://blogs.msdn.microsoft.com/pythonengineering/2016/03/08/python-3-is-winning/ https://blogs.msdn.microsoft.com/pythonengineering/2016/03/0...
- SiVal 10y ago
  Both claims are correct, as is common with statistics. If you write `print "hello"`, you just increased the QUANTITY OF PYTHON 2 CODE. If there is one guy left in the world who occasionally writes some Python 2--and even if he's the only one left who does--the quantity of Python 2 code will, in some sense, "keep on growing", regardless of what happens to Python 3. The article you cite refers to PERCENTAGE OF LIBRARIES that support Python 2 or 3, either one or the other exclusively or both, and it demonstrates a decrease in libraries that only support Python 2 and an increase in libraries that only support Python 3. This is presumably correct, too, but it doesn't suggest that nobody writes any Python 2 code anymore.
Dolores12 10y ago
Can you provide few irreconcilable issues that stops you from using python 3? > Moving it in pieces is also not really possible, since there are many inter-dependencies. I guess something is wrong with your architecture in first place, so you can't separate domains or layers easily.
svisser 10y ago
You can automatically remove six from your codebase after you're Python 3.x only. So six can be a possible solution (provided you have automated tests).
njharman 10y ago
> can't just take 2 months off to move all of it to Python 3 Welcome to the modern COBOL then. Have fun maintaining that and running only old libs cause everyone else in the world has passed you by. > I'm just waiting until a consensus emerges. Wake up, it's done emerged. Almost no libs aren't avail on 3 now.
- NotRustAgain 10y ago
  > Almost no libs aren't avail on 3 now. What an awkward way to say "most libs work on Python 3 now"... Why is that? I suspect it's because saying it clearly doesn't deliver the message you want.
dragonwriter 10y ago
> I have tens of thousands of lines of Python 2 code in a big system. I can't just take 2 months off to move all of it to Python 3. Moving it in pieces is also not really possible, since there are many inter-dependencies. If you cant evolve a system incrementally because of excessive coupling, that's a problem, sure, but not a problem of either the platform you are on or the one you might be thinking about moving to, unless the former forced a tightly-coupled architecture on you.
- tedmiston 10y ago
  We've solved this problem by breaking our architecture up to be more distributed and introducing Python 3 one small service at a time.
  - kstrauser 10y ago
    Same here. It's also been a great opportunity to step back and reevaluate some early-startup engineering decisions that've made the new service codebase fun to work with. Rather than having a single backend that has to handle all possible situations ("this one thing we're talking to speaks XML..."), we can only deploy the stuff actually needed for that one service ("I think we can safely assume all-JSON here"). I highly recommend this approach.

by 21 10y ago

takeda 10y ago

The biggest obstacle people seem to complain about (besides print, which is trivial IMO) is unicode. This is generally also the most painful part of making python 2 code work on python 3. The issue with it is that from unicode perspective your python 2 code is inherently broken. There's no automated tool that can fix it, and there's no way python 5 will be able to run python 2 code correctly unless unicode support would be removed. You have to manually tell python which string is made of characters and which one was made out of bytes.

ash 10y ago

> Me, I wait for Python 5 (=2+3) which will be able to import Python 2 modules, so that you can gradually convert your code to the newer version. Thank you for saying this! For those interested in this direction, there was a discussion about it on HN a couple of months ago: https://news.ycombinator.com/item?id=10823406 https://news.ycombinator.com/item?id=10823406 Proof-of-concept examples embedding Python interpreters within themselves: https://gist.github.com/dutc/eba9b2f7980f400f6287 https://gist.github.com/dutc/eba9b2f7980f400f6287 https://gist.github.com/dutc/2866d969d5e9209d501a https://gist.github.com/dutc/2866d969d5e9209d501a

mixmastamyk 10y ago

> Uglyfying my code with the "six" module it's also not a solution, since when I'll move, I won't care about Python 2 anymore. I've upgraded a lot of my code with just future imports and it is just a tiny step from there, unless you're doing a lot of character encoding. Leave those projects behind, but moving most other projects is easier than I expected.

klodolph 10y ago

All of the reasonable "2+3" features in Python have already landed, like u'' strings in Python 3.3. The really big issues, like the difference between strings and bytes objects, can't really be emulated away. This isn't like WinAmp or MetaTrader where you can just reimplement the old interface in the new application. Code that mixes strings and bytestrings freely is simply not going to interface well with Python 3 code. There are few paths for upgrading. You haven't mentioned 2to3, but I will. I've migraded projects of the same size you mention (10s of kloc) with 2to3 and the process took days, not months. About three days with one developer, actually. There are a number of projects which also have code bases which are compatible with both Python 2 and 3. You say that once you move, you won't care about Python 2 any more, but I that's not a good reason to reject six outright. In my experience, the code isn't really "uglified" but there are just a few minor cases here and there you need to think about. Anyway. Small-ish projects like the one you are talking about are usually not as hard to migrate as you might think. There are some exceptions, of course.

xapata 10y ago
> There are some exceptions, of course. Those exceptions are often bugs in the logic of handling text. Python 2 allows you to sweep that under the rug.
21 10y ago
I agree that the bytes/string problem is a tough one. But the dev team created it and dropped it into the laps of it's users. The Windows API had the same problem, it was ASCII and then they added Unicode. But instead of just forcing your app to exclusively use a single one of them (sort of like use Py2 OR Py3), they allowed you to mix and match your code, and call either the old or the new version of the API. And gradually, people stopped using the older version and started using the Unicode one. Today, new Windows APIs are strictly Unicode. Of course, the solution was ugly from their side, having 2 duplicate APIs for the same thing, but it didn't brought pain to their users and allowed them to proceed at leisure.
- kstrauser 10y ago
  So, it's like Python. You can still mix and match; you just have to explicitly convert between them at the appropriate points. The main breaking change is that you can't read a stream of bytes from a byte-oriented resource (say, a network connection that by definition can only send a stream of octets) and magically treat it like text elsewhere. While that's a tempting thing to want to do, it's been an unending fountain of bugs over the years. For example, in Python 2.7: >>> a = 'this is a test ಠ_ಠ' >>> b = bytes(a) >>> b 'this is a test \xe0\xb2\xa0_\xe0\xb2\xa0' >>> type(b) <type 'str'> In Python 3.5: >>> a = 'this is a test ಠ_ಠ' >>> print(a) this is a test ಠ_ಠ >>> b = bytes(a) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: string argument without an encoding >>> b = bytes(a.encode('utf-8')) >>> b b'this is a test \xe0\xb2\xa0_\xe0\xb2\xa0' >>> type(b) <class 'bytes'> In 2.7, you encode a series of Unicode values back to... a string? This is the source of all kinds of fun as functions expecting a string as input don't have a way to tell whether it's already been encoded or if they need to do so. In 3.5, there's no ambiguity. Functions that write to files or network sockets can either receive strings and know that they have to encode the strings locally, or bytes and know that they don't have to. Right up near the top of Zen we get: >>> import this The Zen of Python, by Tim Peters Beautiful is better than ugly. Explicit is better than implicit. The Python 2-style implicit conversions were awfully convenient 15 years ago when most things were ASCII. They're absolutely painful now that we've moved past the expectation that strings are arrays of single bytes.

syllogism 10y ago

Cython can probably compile your existing codebase such that it's compatible with both 2 and 3. Then you can write your new code in Python 3, while importing the old code as a C extension. You can also do this piece-by-piece, if for some reason your code doesn't compile with Cython right away. The compilation process also makes it much easier to identify bytes/unicode bugs, because you can choose to declare types.

dragonne 10y ago
That sounds like an interesting approach. Have you tried it with a real-world project? I'm curious if the approach has been proven.

luch 10y ago

I've used this module for upgrading a python2-only lib : https://pypi.python.org/pypi/future https://pypi.python.org/pypi/future . Basically it allow you to write Python 3 code that will run on Python 2 interpreter (bar from using advanced features like coroutine or matrix mult operator). Once you decide not to support Python2 anymore, just drop the imports. They also have a Py2/3 compat cheet sheet : http://python-future.org/compatible_idioms.pdf http://python-future.org/compatible_idioms.pdf

daveguy 10y ago
It is pretty obvious there is a problem when your quick and easy cheat sheet is a 33 page tomb. If only Guido and company would acknowledge the problem and fix it instead of putting their fingers in their ears pretending Python 2 will cease to exist in 2020.
- wnoise 10y ago
  They acknowledge the problem, and their solution is to stop actively aiding and abetting the people who are continuing to make it worse by writing python 2 code.
  - nomel 10y ago
    Wow, what a simplification. What about us who have and want to use massive libraries of existing Python 2 code? "Just upgrade your libraries!" is not realistic for everyone, especially those who have to justify a mostly idealogical conversion to a similar language, and spend real time and money, outside of actually developing the end product, to do it. Four more years of official support is a long time.
    - takeda 10y ago
      Someone suggested Cython to compile python 2 code as modules and import to python 3. Did not try it, but that sounds plausible.

ericfrederich 10y ago

It should be noted that the quantity of Python 2 code keeps on growing According to this article Python2 support is shrinking. https://blogs.msdn.microsoft.com/pythonengineering/2016/03/08/python-3-is-winning/ https://blogs.msdn.microsoft.com/pythonengineering/2016/03/0...

SiVal 10y ago
Both claims are correct, as is common with statistics. If you write `print "hello"`, you just increased the QUANTITY OF PYTHON 2 CODE. If there is one guy left in the world who occasionally writes some Python 2--and even if he's the only one left who does--the quantity of Python 2 code will, in some sense, "keep on growing", regardless of what happens to Python 3. The article you cite refers to PERCENTAGE OF LIBRARIES that support Python 2 or 3, either one or the other exclusively or both, and it demonstrates a decrease in libraries that only support Python 2 and an increase in libraries that only support Python 3. This is presumably correct, too, but it doesn't suggest that nobody writes any Python 2 code anymore.

Dolores12 10y ago

Can you provide few irreconcilable issues that stops you from using python 3? > Moving it in pieces is also not really possible, since there are many inter-dependencies. I guess something is wrong with your architecture in first place, so you can't separate domains or layers easily.

svisser 10y ago

You can automatically remove six from your codebase after you're Python 3.x only. So six can be a possible solution (provided you have automated tests).

njharman 10y ago

> can't just take 2 months off to move all of it to Python 3 Welcome to the modern COBOL then. Have fun maintaining that and running only old libs cause everyone else in the world has passed you by. > I'm just waiting until a consensus emerges. Wake up, it's done emerged. Almost no libs aren't avail on 3 now.

NotRustAgain 10y ago
> Almost no libs aren't avail on 3 now. What an awkward way to say "most libs work on Python 3 now"... Why is that? I suspect it's because saying it clearly doesn't deliver the message you want.

dragonwriter 10y ago

> I have tens of thousands of lines of Python 2 code in a big system. I can't just take 2 months off to move all of it to Python 3. Moving it in pieces is also not really possible, since there are many inter-dependencies. If you cant evolve a system incrementally because of excessive coupling, that's a problem, sure, but not a problem of either the platform you are on or the one you might be thinking about moving to, unless the former forced a tightly-coupled architecture on you.

tedmiston 10y ago
We've solved this problem by breaking our architecture up to be more distributed and introducing Python 3 one small service at a time.
- kstrauser 10y ago
  Same here. It's also been a great opportunity to step back and reevaluate some early-startup engineering decisions that've made the new service codebase fun to work with. Rather than having a single backend that has to handle all possible situations ("this one thing we're talking to speaks XML..."), we can only deploy the stuff actually needed for that one service ("I think we can safely assume all-JSON here"). I highly recommend this approach.