5 ms·

This is a good question, and I think about it as well. My best guess for a simple explanation: Python is very popular; it makes sense to improve performance for

This is a good question, and I think about it as well. My best guess for a simple explanation: Python is very popular; it makes sense to improve performance for python users, given many do not wish to learn to use a more performant language, or to use a more performant Python implementation. Becoming proficient in a range of tools so you can use the right one for the right job is high enough friction that it is not the path chosen by many.

eigenspace 2y ago
Oh yeah, I totally get the motivation behind it. It's always very tempting to want to make things faster. But I can't help but wondering if these attempts to make it faster might end up just making it worse. On the other hand though, Python is so big and there's so many corps using it with so much cash that maybe they can get away with just breaking shit every few releases and people will just go adapt packages to the changes.
- dagmx 2y ago
  Python famously has a community that does NOT adapt to changes well. See the Python 2 to 3 transition.
  - eigenspace 2y ago
    Python's community was significantly smaller and less flushed with cash during the 2 to 3 transition. Since then there has been numerous 3.x releases that were breaking and people seem to have been sucking it up and dealing with it quietly so far. The main thing is that unlike the 2 to 3 transition, they're not breaking syntax (for the most part?), which everyone experiences and has an opinion on, they're breaking rather deep down things that for the most part only the big packages rely on so most users don't experience it much at all.
    - dagmx 2y ago
      I disagree with this entire comment. The Python community consisted of tons of developers including very wealthy companies. At what point in the last few years would you even say they became “rich enough” to do the migration? Because people are STILL talking about trying to fork 2.7 into a 2.8. I also disagree with your assertion that 3.x releases have significant breaking changes. Could you point to any specific major breaking changes between 3.x releases? 2 to 3 didn’t break syntax for most code either. It largely cleaned house on sensible API defaults.
      - eigenspace 2y ago
        Fair enough. You may be totally right here, as I mentioned I don't use Python much at all since like 2017 and haven't paid it much attention in a while. I retract my comment. Regarding breakage in 3.x, all I know is that I recall several times where I did a linux system update (rolling release), and that updated my Python to a newly released version which broke various things in my system. I'm pretty sure one of these was v3.10, but I forget which others caused me problems which I could only solve by pinning Python to an older release. It's entirely possible though that no actual APIs were broken and that this was just accidentaly bugs in the release, or the packages were being naughty and relying on internals they shouldn't have relied on or something else.
        dagmx 2y ago
        To your last point: it’s neither the language nor the packages but rather it’s the ABI. Python isn’t fully ABI stable (though it’s improved greatly) so you can’t just intermix compiled dependencies between different versions of Python. This is true for many packages in your distro as well.
        eigenspace 2y ago
        Ah I see. Yeah, I guess I just think that for a language that's so dependent on FFI, instability in the ABI is defacto instability in the language as far as I'm concerned. But I understand that not everyone feels the same. Almost every major noteworthy Python package uses the ABI, so instability there is going to constantly be felt ecosystem wide.
  - Yossarrian22 2y ago
    Are there communities that handle such a change well? At least that went better than Perl and Raku
    - dagmx 2y ago
      Anything where the language frontend isn’t tied to the ABI compatibility of the artifacts I think. They can mix versions/editions without worry. I think it’s a larger problem with interpreted languages where all the source has to be in a single version. In that case I cant think of much.
  - adamc 2y ago
    That was, in many ways, a crazy difficult transition. I don't think most languages have gone through such a thing. Perl tried and died. So I don't agree that it reflects poorly on the community; I think the plan itself was too ambitious.
    - dagmx 2y ago
      Many languages have. There were significant breaks in C++ when stringabi changed, Swift has had major language changes, rust has editions. The difference is in what motivates getting to the other end of that transition bump and how big the bump is. That’s why it took till past 2.7’s EOL to actually get people on to 3 in a big way because they’d drag their feet if they don’t see a big enough change. Compiled languages have it easier because they don’t need to mix source between dependencies, they just have to be ABI compatible.
- lanstin 2y ago
  I think I am much happier to redo my personal scripts to stay at bleeding edge than to rewrite old code at work to stay in supported versions. For corporations wanton breaking changes means code that is working and has some non-zero risk to change has to have budget spent on it just to stay in place. Just this week I failed to convince a team to migrate their ten year old maintenance mode demon to Python 3.
devjab 2y ago
You should really add that Python is also a very good tool for people who know more performant languages. I think one of the sides which often gets forgotten is that a lot of software will never actually need to be very performant and often you’re not going to know the bottlenecks beforehand. If you even get to the bottlenecks it means you’ve succeeded enough to get to the bottlenecks. Somewhere you might not have gotten if you over engineered things before you needed it. What makes Python brilliant is that it’s easy to deliver on business needs. It’s easy to include people who aren’t actually software engineers but can write Python to do their stuff. It’s easy to make that Wild West code sane. Most importantly, however, it’s extremely easy to replace parts of your Python code with something like C (or Zig). So even if you know performant languages, you can still use Python for most things and then as glue for heavy computation. Now I may have made it sound like I think Python is brilliant so I’d like to add that I actually think it’s absolute trash. Loveable trash.
- nomel 2y ago
  > it’s extremely easy to replace parts of your Python code with something like C I tend to use C++, so use SWIG [1] to make python code to interface with C++ (or C). You can nearly just give it a header file, and a python class pops out, with native types and interfaces. It's really magical. [1] https://www.swig.org https://www.swig.org
- lanstin 2y ago
  I do think not being able to use 32 cores easily is a gap in the current language. In 2017 I rewrote a fairly high performance python dialog to Kafka daemon in Go. The python version require a lot of specialized knowledge to write, using gevent, PyPy, hand optimizing our framework for the hot paths etc, and still was only using a few cores to do work. The Go was a dead simple my first Go project sort of implementation and used 32 cores and therefore worked much better right out of the gate. (I mean I did have go routine worker pools for each step of the processing, but the division of work into the stages was already in the Python code). So yeah Python is easy to make less of a mess of for lots of people, until you want to use all your cores (which again means rerunning your things goes from four minutes to 15 seconds on that fancy laptop).

eigenspace 2y ago

Oh yeah, I totally get the motivation behind it. It's always very tempting to want to make things faster. But I can't help but wondering if these attempts to make it faster might end up just making it worse. On the other hand though, Python is so big and there's so many corps using it with so much cash that maybe they can get away with just breaking shit every few releases and people will just go adapt packages to the changes.

dagmx 2y ago
Python famously has a community that does NOT adapt to changes well. See the Python 2 to 3 transition.
- eigenspace 2y ago
  Python's community was significantly smaller and less flushed with cash during the 2 to 3 transition. Since then there has been numerous 3.x releases that were breaking and people seem to have been sucking it up and dealing with it quietly so far. The main thing is that unlike the 2 to 3 transition, they're not breaking syntax (for the most part?), which everyone experiences and has an opinion on, they're breaking rather deep down things that for the most part only the big packages rely on so most users don't experience it much at all.
  - dagmx 2y ago
    I disagree with this entire comment. The Python community consisted of tons of developers including very wealthy companies. At what point in the last few years would you even say they became “rich enough” to do the migration? Because people are STILL talking about trying to fork 2.7 into a 2.8. I also disagree with your assertion that 3.x releases have significant breaking changes. Could you point to any specific major breaking changes between 3.x releases? 2 to 3 didn’t break syntax for most code either. It largely cleaned house on sensible API defaults.
    - eigenspace 2y ago
      Fair enough. You may be totally right here, as I mentioned I don't use Python much at all since like 2017 and haven't paid it much attention in a while. I retract my comment. Regarding breakage in 3.x, all I know is that I recall several times where I did a linux system update (rolling release), and that updated my Python to a newly released version which broke various things in my system. I'm pretty sure one of these was v3.10, but I forget which others caused me problems which I could only solve by pinning Python to an older release. It's entirely possible though that no actual APIs were broken and that this was just accidentaly bugs in the release, or the packages were being naughty and relying on internals they shouldn't have relied on or something else.
      - dagmx 2y ago
        To your last point: it’s neither the language nor the packages but rather it’s the ABI. Python isn’t fully ABI stable (though it’s improved greatly) so you can’t just intermix compiled dependencies between different versions of Python. This is true for many packages in your distro as well.
        eigenspace 2y ago
        Ah I see. Yeah, I guess I just think that for a language that's so dependent on FFI, instability in the ABI is defacto instability in the language as far as I'm concerned. But I understand that not everyone feels the same. Almost every major noteworthy Python package uses the ABI, so instability there is going to constantly be felt ecosystem wide.
- Yossarrian22 2y ago
  Are there communities that handle such a change well? At least that went better than Perl and Raku
  - dagmx 2y ago
    Anything where the language frontend isn’t tied to the ABI compatibility of the artifacts I think. They can mix versions/editions without worry. I think it’s a larger problem with interpreted languages where all the source has to be in a single version. In that case I cant think of much.
- adamc 2y ago
  That was, in many ways, a crazy difficult transition. I don't think most languages have gone through such a thing. Perl tried and died. So I don't agree that it reflects poorly on the community; I think the plan itself was too ambitious.
  - dagmx 2y ago
    Many languages have. There were significant breaks in C++ when stringabi changed, Swift has had major language changes, rust has editions. The difference is in what motivates getting to the other end of that transition bump and how big the bump is. That’s why it took till past 2.7’s EOL to actually get people on to 3 in a big way because they’d drag their feet if they don’t see a big enough change. Compiled languages have it easier because they don’t need to mix source between dependencies, they just have to be ABI compatible.
lanstin 2y ago
I think I am much happier to redo my personal scripts to stay at bleeding edge than to rewrite old code at work to stay in supported versions. For corporations wanton breaking changes means code that is working and has some non-zero risk to change has to have budget spent on it just to stay in place. Just this week I failed to convince a team to migrate their ten year old maintenance mode demon to Python 3.

devjab 2y ago

You should really add that Python is also a very good tool for people who know more performant languages. I think one of the sides which often gets forgotten is that a lot of software will never actually need to be very performant and often you’re not going to know the bottlenecks beforehand. If you even get to the bottlenecks it means you’ve succeeded enough to get to the bottlenecks. Somewhere you might not have gotten if you over engineered things before you needed it. What makes Python brilliant is that it’s easy to deliver on business needs. It’s easy to include people who aren’t actually software engineers but can write Python to do their stuff. It’s easy to make that Wild West code sane. Most importantly, however, it’s extremely easy to replace parts of your Python code with something like C (or Zig). So even if you know performant languages, you can still use Python for most things and then as glue for heavy computation. Now I may have made it sound like I think Python is brilliant so I’d like to add that I actually think it’s absolute trash. Loveable trash.

nomel 2y ago
> it’s extremely easy to replace parts of your Python code with something like C I tend to use C++, so use SWIG [1] to make python code to interface with C++ (or C). You can nearly just give it a header file, and a python class pops out, with native types and interfaces. It's really magical. [1] https://www.swig.org https://www.swig.org
lanstin 2y ago
I do think not being able to use 32 cores easily is a gap in the current language. In 2017 I rewrote a fairly high performance python dialog to Kafka daemon in Go. The python version require a lot of specialized knowledge to write, using gevent, PyPy, hand optimizing our framework for the hot paths etc, and still was only using a few cores to do work. The Go was a dead simple my first Go project sort of implementation and used 32 cores and therefore worked much better right out of the gate. (I mean I did have go routine worker pools for each step of the processing, but the division of work into the stages was already in the Python code). So yeah Python is easy to make less of a mess of for lots of people, until you want to use all your cores (which again means rerunning your things goes from four minutes to 15 seconds on that fancy laptop).