6 ms·

I can't help but believe a sane world would've moved from Python to PyPy Yes, and this is my fundamental complaint with the Python 3 transition. It took probab

I can't help but believe a sane world would've moved from Python to PyPy Yes, and this is my fundamental complaint with the Python 3 transition. It took probably millions of engineer-hours, and the result was better Unicode support and a handful of minor features most of which could have been added to Python 2 just as easily. I suspect most users would gladly have traded those benefits for a 10x performance improvement.

mehrdadn 7y ago
I didn't want to reply to the PyPy comment and be negative, but I haven't really gotten speedup from the few times I've tried PyPy. In fact I've generally gotten slowdowns. Definitely not 10x improvements. I'm not sure what the reason might be though, because the rest of the world seems to think differently.
- ericflo 7y ago
  It's been a while since I used PyPy, but JITs in general warm up over time. If you have hot loops with heavy arithmetic and no branches, that's usually best case scenario for JITs. If you have branchy non-uniform control flow, that's the worse scenario. So it really depends on your usage - you may be paying the JIT overhead costs with little benefit.
  - xscott 7y ago
    Spot on, and the difference between a tracing JIT vs a method JIT can be night and day too.
- ehsankia 7y ago
  Really depends on the kind of work you're doing. As mentioned above, numerical and looping code will see the most benefit. I used it heavily when doing Project Euler problems, and those would easily see 10-100x speedups, taking some problems from 5-10m run time to seconds.
xscott 7y ago
> millions of engineer-hours, and the result was better Unicode support and a handful of minor features Yes, and I wish the better Unicode support had been implemented similarly to how Go does it - one string type, and you use it to hold UTF-8 if needed. In other words, they could've simply deprecated the Python 2.X unicode object and added libraries to extract code points, or grapheme clusters, or perform normalization etc... This seems much simpler and more "Pythonic". I guess everything is 20/20 in hindsight.
- suzuki 7y ago
  I totally agree with you. I hope we still have a chance to do it well with PyPy.
scoot_718 7y ago
> and the result was better Unicode support Different Unicode support. And worse bytes support. What could previously be done using python -c "..." is now long, horrible and ugly.
- mehrdadn 7y ago
  > Different Unicode support. And worse bytes support. I feel like you're the first person I've seen on the planet to echo my sentiments on these. I expect a lot of people will jump here to tell you you're wrong like they have to me, so just wanted to let you know I've felt exactly these pains and agree with you.
  - harikb 7y ago
    I am hoping this is in agreement. py2’s flexibility to handle utf8 bytes without fuss is amazing. Then people come up with all kind of purity reasons to make it more complicated.
    - mehrdadn 7y ago
      Take out "utf8" and I'll agree ;) The fundamental problem as I see it is that "string" is a grossly leaky and misunderstood abstraction. The string type is not the same thing as a "text" type. It's being used in all the wrong places for that purpose. People treat "string" like it means "text", but in so many places where we deal with them, they just aren't (and should never be) text. Everything from stdio to argv to file paths to environment variables to "text" files to basically any interface with the outside world needs to be dealt with in bytes rather than text if you care about actually producing correct code that doesn't lose, corrupt, or otherwise choke on data. C++ understood this and got it right, preferring to focus on optimizing rather than constraining the string type. Many other languages did pretty well by avoiding enforcing encodings on strings, too. And Python 2 defaulted to bytes as well, and only really cared about encoding/decoding at I/O boundaries where it thought it can assume it's dealing with text (though it sometimes didn't behave well there, and yes it got painful as a result). Then Python 3 came along and just made everyone start treating most data as if they're inherently (Unicode) text by default, when they really had no such constraints to begin with. It boggles my mind that Python 3 folks like to beat the drum on how Python 3 got the bytes/unicode right without taking a single moment to even notice that most strings people deal with aren't (and never were!) actually guaranteed to be in a specific, known textual encoding a priori. They were just arrays of code units with few restrictions on them, and if you want to write correct code, you're going to have to deal with bytes by default (or something else with similar flexibility) instead of text. It would've been totally fine to introduce a text type, but it fundamentally can't take the place of a blob type, which is the language of the outside world.
CydeWeys 7y ago
Yup. I like Python3, it's better than 2, and has lots of good new features, but so many of them could simply have been added onto 2 without requiring this huge painstaking migration that cost an unbelievable amount of effort worldwide. Contrast with Java, which has made much more substantial changes to the language over all these years, but goes to great pains to support backwards compatibility. Upgrading major Java versions was never remotely as painful, and as a result, huge amounts of engineering time was saved.
- takeda 7y ago
  Python 3 features in Python 2 was entire purpose of Python 2.7. All features it came with were backports of python 3 features. They could continue doing that, but then we would never fix the Unicode. There's so much whining, when they gave over 5 years to do the migration. I think the whole problem was that they gave too much time, and people were thinking it will be like that forever. Also, Java is not a dynamic language, the type system allows for easy fix of issues like these, the python solution to do that is mypy, but it requires work by adding types.
  - philwelch 7y ago
    > There's so much whining, when they gave over 5 years to do the migration. You make it sound so generous of the Python maintainers to "give over five years" before breaking backwards compatibility--something that C, Java, JavaScript, C++, and probably most other languages haven't done for decades, or in some cases ever. Programming languages are serious basic infrastructure, programmer time is valuable, and Python's installed base is large enough that many man-centuries were probably wasted on this migration. If the Python maintainers had not given as much time as they did, I doubt the Python community or ecosystem would have transitioned any more quickly than they did. More likely would have been some fork or alternate implementation of Python 2 becoming the new standard.
    - kerkeslager 7y ago
      Okay, but how many man-centuries have we spent fixing buffer overflows in C because the type system can't check that without breaking reverse compatibility? Reverse compatibility isn't free.
fnord123 7y ago
It's only 10x for web. Anyone using python for batch processing won't see the gains because the work is done before pypy warms up. But yes the only people who wanted better unicode were web people. So they could have been better served moving to pypy.
rstuart4133 7y ago
> better Unicode support I guess it's a matter of emphasis, but I'd say it has different Unicode support. It's better mildly for some use cases, but worse for others. It's bloody horrible for one use case in particular: when you know the text is readable and mostly ASCII based and you are only interested in the ASCII bits, but don't know the encoding. That is the position you find yourself in for any designed in pre-unicode times, and that happens to include just about every file in a Unix file system. The solution in those circumstances is to treat everything as bytes (b''). That wasn't even possible in the beginning. Now it mostly works, but all with hundreds of corner exceptions (like Exception messages, so you can't easily include a Unix filename in an error message).

mehrdadn 7y ago

I didn't want to reply to the PyPy comment and be negative, but I haven't really gotten speedup from the few times I've tried PyPy. In fact I've generally gotten slowdowns. Definitely not 10x improvements. I'm not sure what the reason might be though, because the rest of the world seems to think differently.

ericflo 7y ago
It's been a while since I used PyPy, but JITs in general warm up over time. If you have hot loops with heavy arithmetic and no branches, that's usually best case scenario for JITs. If you have branchy non-uniform control flow, that's the worse scenario. So it really depends on your usage - you may be paying the JIT overhead costs with little benefit.
- xscott 7y ago
  Spot on, and the difference between a tracing JIT vs a method JIT can be night and day too.
ehsankia 7y ago
Really depends on the kind of work you're doing. As mentioned above, numerical and looping code will see the most benefit. I used it heavily when doing Project Euler problems, and those would easily see 10-100x speedups, taking some problems from 5-10m run time to seconds.

xscott 7y ago

> millions of engineer-hours, and the result was better Unicode support and a handful of minor features Yes, and I wish the better Unicode support had been implemented similarly to how Go does it - one string type, and you use it to hold UTF-8 if needed. In other words, they could've simply deprecated the Python 2.X unicode object and added libraries to extract code points, or grapheme clusters, or perform normalization etc... This seems much simpler and more "Pythonic". I guess everything is 20/20 in hindsight.

suzuki 7y ago
I totally agree with you. I hope we still have a chance to do it well with PyPy.

scoot_718 7y ago

> and the result was better Unicode support Different Unicode support. And worse bytes support. What could previously be done using python -c "..." is now long, horrible and ugly.

mehrdadn 7y ago
> Different Unicode support. And worse bytes support. I feel like you're the first person I've seen on the planet to echo my sentiments on these. I expect a lot of people will jump here to tell you you're wrong like they have to me, so just wanted to let you know I've felt exactly these pains and agree with you.
- harikb 7y ago
  I am hoping this is in agreement. py2’s flexibility to handle utf8 bytes without fuss is amazing. Then people come up with all kind of purity reasons to make it more complicated.
  - mehrdadn 7y ago
    Take out "utf8" and I'll agree ;) The fundamental problem as I see it is that "string" is a grossly leaky and misunderstood abstraction. The string type is not the same thing as a "text" type. It's being used in all the wrong places for that purpose. People treat "string" like it means "text", but in so many places where we deal with them, they just aren't (and should never be) text. Everything from stdio to argv to file paths to environment variables to "text" files to basically any interface with the outside world needs to be dealt with in bytes rather than text if you care about actually producing correct code that doesn't lose, corrupt, or otherwise choke on data. C++ understood this and got it right, preferring to focus on optimizing rather than constraining the string type. Many other languages did pretty well by avoiding enforcing encodings on strings, too. And Python 2 defaulted to bytes as well, and only really cared about encoding/decoding at I/O boundaries where it thought it can assume it's dealing with text (though it sometimes didn't behave well there, and yes it got painful as a result). Then Python 3 came along and just made everyone start treating most data as if they're inherently (Unicode) text by default, when they really had no such constraints to begin with. It boggles my mind that Python 3 folks like to beat the drum on how Python 3 got the bytes/unicode right without taking a single moment to even notice that most strings people deal with aren't (and never were!) actually guaranteed to be in a specific, known textual encoding a priori. They were just arrays of code units with few restrictions on them, and if you want to write correct code, you're going to have to deal with bytes by default (or something else with similar flexibility) instead of text. It would've been totally fine to introduce a text type, but it fundamentally can't take the place of a blob type, which is the language of the outside world.

CydeWeys 7y ago

Yup. I like Python3, it's better than 2, and has lots of good new features, but so many of them could simply have been added onto 2 without requiring this huge painstaking migration that cost an unbelievable amount of effort worldwide. Contrast with Java, which has made much more substantial changes to the language over all these years, but goes to great pains to support backwards compatibility. Upgrading major Java versions was never remotely as painful, and as a result, huge amounts of engineering time was saved.

takeda 7y ago
Python 3 features in Python 2 was entire purpose of Python 2.7. All features it came with were backports of python 3 features. They could continue doing that, but then we would never fix the Unicode. There's so much whining, when they gave over 5 years to do the migration. I think the whole problem was that they gave too much time, and people were thinking it will be like that forever. Also, Java is not a dynamic language, the type system allows for easy fix of issues like these, the python solution to do that is mypy, but it requires work by adding types.
- philwelch 7y ago
  > There's so much whining, when they gave over 5 years to do the migration. You make it sound so generous of the Python maintainers to "give over five years" before breaking backwards compatibility--something that C, Java, JavaScript, C++, and probably most other languages haven't done for decades, or in some cases ever. Programming languages are serious basic infrastructure, programmer time is valuable, and Python's installed base is large enough that many man-centuries were probably wasted on this migration. If the Python maintainers had not given as much time as they did, I doubt the Python community or ecosystem would have transitioned any more quickly than they did. More likely would have been some fork or alternate implementation of Python 2 becoming the new standard.
  - kerkeslager 7y ago
    Okay, but how many man-centuries have we spent fixing buffer overflows in C because the type system can't check that without breaking reverse compatibility? Reverse compatibility isn't free.

fnord123 7y ago

It's only 10x for web. Anyone using python for batch processing won't see the gains because the work is done before pypy warms up. But yes the only people who wanted better unicode were web people. So they could have been better served moving to pypy.

rstuart4133 7y ago

> better Unicode support I guess it's a matter of emphasis, but I'd say it has different Unicode support. It's better mildly for some use cases, but worse for others. It's bloody horrible for one use case in particular: when you know the text is readable and mostly ASCII based and you are only interested in the ASCII bits, but don't know the encoding. That is the position you find yourself in for any designed in pre-unicode times, and that happens to include just about every file in a Unix file system. The solution in those circumstances is to treat everything as bytes (b''). That wasn't even possible in the beginning. Now it mostly works, but all with hundreds of corner exceptions (like Exception messages, so you can't easily include a Unix filename in an error message).