3 ms·

The entire Python community was in pain over Python 3 for 10 years, even if migrating any particular program wasn't much trouble. If you want to contest the no

The entire Python community was in pain over Python 3 for 10 years, even if migrating any particular program wasn't much trouble. If you want to contest the notion that there was pain, then fine: most of the community simply ignored Python 3 for 10 years, because there was no reason until quite late in the process to worry about it. I myself never bothered migrating any of my Python 2 stuff. It might not be difficult to do so, but continuing to run it under Python 2 still works fine. If you migrated all of yours in 2 hours, you must not have had much to start with. I do use Python 3 for new stuff most of the time by now, but I keep running into little snags, like the .decode() method not working on strings, or having some (but not all) of the codecs removed from the string module so you have to use the codecs module. There's also the matter of stuff that is supposedly ported but isn't completely. For example, Beautiful Soup works nicely under py3, but it doesn't have a typeshed entry so its import needs a special annotation to stop mypy from complaining about it. The real loss with Python 3 is that it could have been so much better than it is. I remember hearing that Go expected to pick up a lot of migrating C++ users, but it got migrating Python users instead. Here's a pain story about a 2 to 3 migration though: https://dropbox.tech/application/how-we-rolled-out-one-of-the-largest-python-3-migrations-ever https://dropbox.tech/application/how-we-rolled-out-one-of-th...

dagenix 5y ago
IMO, it's a good thing that decode() doesn't on strings in Python 3.
- throwaway81523 5y ago
  There is a case to be made either way. The point is that it's yet another quirky difference that pops up sometimes, and using python3 keeps bringing up a stream of such issues. It does seem in the spirit of Python's duck typing to be able to say '616263'.decode('hex') and get 'abc', and that does work fine in py2. Try it in py3 and you get a type error. So ok, convert the string to bytes, e.g. by saying b'616263' instead of '616263'. That doesn't work either, but I'll leave the experiment to you.
  - adwn 5y ago
    So, like this? codecs.decode(b'616263', 'hex') If that's correct, what's so hard/bad about this?
__turbobrew__ 5y ago
Some people don’t get that migrating large legacy code bases from 2 to 3 is not trivial. Your dropbox example is a good one: the migration took 3 years to complete. What else could dropbox have achieved with a 3 year long project if they didn’t have to migrate from 2 to 3?
- throwaway81523 5y ago
  And Guido van Rossum himself was working at dropbox at the time, contributing to that port. Imagine how much harder it might have been without him.
zzzeek 5y ago
And yet python tops the Tiobe index anyway. The python 3 transition was not that big a deal, and I migrated or assisted in the migration of many codebases some of which are as big as they get (openstack). the ten year thing especially made it really gradual, and we're all done now. Python 3 is great. Folks migrating to rust / go etc are looking at performance concerns for high volume server software, which python is simply never going to specialize in, and for which rust and go were both introduced only very recently. They didn't switch because python 3 was so disappointing.
- tialaramex 5y ago
  However if you're confident Python simply is "never going to specialize in" performance, maybe the GIL isn't a problem and this work is irrelevant right? I'm dubious about this work because I think it tries to paper over the cracks in code with data races, but maybe the problem was there already with the GIL and this changes little.
  - zzzeek 5y ago
    I agree, the GIL is really not a huge problem unless you are trying to write a high frequency trading app in Python, which you shouldn't (and not just for technical reasons ;) )
- jgehrcke 5y ago
  By the way Mike I think https://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ https://techspot.zzzeek.org/2015/02/15/asynchronous-python-a... is still one of the best articles on the topic of "python and concurrency". So many important insights for real-world engineering. Thanks for all your work in the Python ecosystem!
rtpg 5y ago
`.decode` doesn't work on strings because you decode bytes! PYthon has UTF strings! I agree about the annoyance of the codec modules, and stuff like the urllib reorg happening on 3.0 instead of shifting it to later releases (and self-owns like the u prefix). But if you're calling `decode` on a string you have a bug in hiding, from the moment that your "code that handles encodings transparently" blows up when someone gives you a file that isn't UTF-encoded. signed: someone who did the 2 -> 3 transition for a project that deals with a lot of non-UTF-stuff.
- garmaine 5y ago
  So? `.decode(“hex”)` still makes sense on a string, and was >95% of the encode/decode calls I had.
  - rtpg 5y ago
    Hex-encoding works off of bytes… like if a character has a different byte representation between two encodings you need to specify in both directions Even the Python 2 strategy for hex might lead to your text getting garbled if you’re using anything not représentable in ASCII (roughly).
    - garmaine 5y ago
      That’s not what decode(“hex”) does though. It takes strings in whatever encoding and turns 0-9,a-f into bytes. In Python 3 it would presumably return a bytes object instead, which you could then hypothetically call encode(“hex”) on. But the point is that it would have kept the syntax the same even if the underlying types changed. Python3 unnecessarily broke compatibility in many places. Yes I agree that using bytes.fromhex() and .hex() are better design choices. But requiring manual inspection and updating of all strong manipulation code for the transition was an outrageous requirement, particularly for a dynamic language that doesn’t (or at least didn’t at that time) have compile-time type checking to catch such errors.
BeetleB 5y ago
> The entire Python community was in pain over Python 3 for 10 years Sorry, but that's fiction. Yes, lots of projects (still the minority) simply didn't upgrade, but it's not because they tried and failed. It's because they never prioritized it. > Here's a pain story about a 2 to 3 migration though: Consistent with my claim: > Probably less than 1% of projects had serious issues with the migration. We just hear of a few popular ones that had some pain and assume that was representative.
- snovv_crash 5y ago
  Lots of things didn't upgrade because their dependencies didn't. And the dependencies had no reason to, because 3.0 didn't offer anything they didn't have before.
  - BeetleB 5y ago
    Quite a bit before the deadline, pretty much all dependencies had upgraded. Those that hadn't were more or less abandoned projects. Sure, you could hunt and find some that were still maintained/developed with Python 2 in, say, 2019. But these again will be the tiny minority. I still stand by my 1% claim. The problems with upgrading have been vastly exaggerated.
    - JanisL 5y ago
      The pain in the Python 2->3 conversion I found came more from in house packages being written in Python 2 rather than public packages.

dagenix 5y ago

IMO, it's a good thing that decode() doesn't on strings in Python 3.

throwaway81523 5y ago
There is a case to be made either way. The point is that it's yet another quirky difference that pops up sometimes, and using python3 keeps bringing up a stream of such issues. It does seem in the spirit of Python's duck typing to be able to say '616263'.decode('hex') and get 'abc', and that does work fine in py2. Try it in py3 and you get a type error. So ok, convert the string to bytes, e.g. by saying b'616263' instead of '616263'. That doesn't work either, but I'll leave the experiment to you.
- adwn 5y ago
  So, like this? codecs.decode(b'616263', 'hex') If that's correct, what's so hard/bad about this?

__turbobrew__ 5y ago

Some people don’t get that migrating large legacy code bases from 2 to 3 is not trivial. Your dropbox example is a good one: the migration took 3 years to complete. What else could dropbox have achieved with a 3 year long project if they didn’t have to migrate from 2 to 3?

throwaway81523 5y ago
And Guido van Rossum himself was working at dropbox at the time, contributing to that port. Imagine how much harder it might have been without him.

zzzeek 5y ago

And yet python tops the Tiobe index anyway. The python 3 transition was not that big a deal, and I migrated or assisted in the migration of many codebases some of which are as big as they get (openstack). the ten year thing especially made it really gradual, and we're all done now. Python 3 is great. Folks migrating to rust / go etc are looking at performance concerns for high volume server software, which python is simply never going to specialize in, and for which rust and go were both introduced only very recently. They didn't switch because python 3 was so disappointing.

tialaramex 5y ago
However if you're confident Python simply is "never going to specialize in" performance, maybe the GIL isn't a problem and this work is irrelevant right? I'm dubious about this work because I think it tries to paper over the cracks in code with data races, but maybe the problem was there already with the GIL and this changes little.
- zzzeek 5y ago
  I agree, the GIL is really not a huge problem unless you are trying to write a high frequency trading app in Python, which you shouldn't (and not just for technical reasons ;) )
jgehrcke 5y ago
By the way Mike I think https://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ https://techspot.zzzeek.org/2015/02/15/asynchronous-python-a... is still one of the best articles on the topic of "python and concurrency". So many important insights for real-world engineering. Thanks for all your work in the Python ecosystem!

rtpg 5y ago

`.decode` doesn't work on strings because you decode bytes! PYthon has UTF strings! I agree about the annoyance of the codec modules, and stuff like the urllib reorg happening on 3.0 instead of shifting it to later releases (and self-owns like the u prefix). But if you're calling `decode` on a string you have a bug in hiding, from the moment that your "code that handles encodings transparently" blows up when someone gives you a file that isn't UTF-encoded. signed: someone who did the 2 -> 3 transition for a project that deals with a lot of non-UTF-stuff.

garmaine 5y ago
So? `.decode(“hex”)` still makes sense on a string, and was >95% of the encode/decode calls I had.
- rtpg 5y ago
  Hex-encoding works off of bytes… like if a character has a different byte representation between two encodings you need to specify in both directions Even the Python 2 strategy for hex might lead to your text getting garbled if you’re using anything not représentable in ASCII (roughly).
  - garmaine 5y ago
    That’s not what decode(“hex”) does though. It takes strings in whatever encoding and turns 0-9,a-f into bytes. In Python 3 it would presumably return a bytes object instead, which you could then hypothetically call encode(“hex”) on. But the point is that it would have kept the syntax the same even if the underlying types changed. Python3 unnecessarily broke compatibility in many places. Yes I agree that using bytes.fromhex() and .hex() are better design choices. But requiring manual inspection and updating of all strong manipulation code for the transition was an outrageous requirement, particularly for a dynamic language that doesn’t (or at least didn’t at that time) have compile-time type checking to catch such errors.

BeetleB 5y ago

> The entire Python community was in pain over Python 3 for 10 years Sorry, but that's fiction. Yes, lots of projects (still the minority) simply didn't upgrade, but it's not because they tried and failed. It's because they never prioritized it. > Here's a pain story about a 2 to 3 migration though: Consistent with my claim: > Probably less than 1% of projects had serious issues with the migration. We just hear of a few popular ones that had some pain and assume that was representative.

snovv_crash 5y ago
Lots of things didn't upgrade because their dependencies didn't. And the dependencies had no reason to, because 3.0 didn't offer anything they didn't have before.
- BeetleB 5y ago
  Quite a bit before the deadline, pretty much all dependencies had upgraded. Those that hadn't were more or less abandoned projects. Sure, you could hunt and find some that were still maintained/developed with Python 2 in, say, 2019. But these again will be the tiny minority. I still stand by my 1% claim. The problems with upgrading have been vastly exaggerated.
  - JanisL 5y ago
    The pain in the Python 2->3 conversion I found came more from in house packages being written in Python 2 rather than public packages.