5 ms·

My use of python is somewhat recent. But the two languages that I have used a lot of - Java and JS - have interpreters that were heavily optimized over time. I

by sieve 2y ago

My use of python is somewhat recent. But the two languages that I have used a lot of - Java and JS - have interpreters that were heavily optimized over time. I wonder why that never happened with python and, instead, everyone continues to write their critical code in C/Rust. I am planning to shift some of my stuff to pypy (so a "fast" python exists, kind of). But some dependencies can be problematic, I have heard.

Numerlor 2y ago
python just didn't have much momentum until relatively recently, despite it's age. There are efforts to speed it up going on now backed by Microsoft. For pypy it's in a weird spot as the things it does fast are the ones you'd usually just offload to a module implemented in C
- sieve 2y ago
  > There are efforts to speed it up Well, the extensions are going to complicate this a lot. A fast JIT interpreter cannot reach into a library and do its magic there the way HotSpot/V8 can with native Java/JS code.
- simonw 2y ago
  As a long time Pythonista I was going to push back against your suggestion that Python didn't have much momentum until recently, but then I looked at the historic graph on https://www.tiobe.com/tiobe-index/ https://www.tiobe.com/tiobe-index/ and yeah, Python's current huge rise in popularity didn't really get started until around 2018. (TIOBE's methodology is a bit questionable though, as far as I can tell it's almost entirely based on how many search engine hits they get for "X programming". https://www.tiobe.com/tiobe-index/programminglanguages_definition/ https://www.tiobe.com/tiobe-index/programminglanguages_defin...)
  - tialaramex 2y ago
    Yes, TIOBE is garbage. The biggest problem is that because they're coy about methodology we don't even know what we're talking about. Rust's "Most Loved" Stack Overflow numbers were at least a specific thing where you can say OK that doesn't mean there's more Rust software or that Rust programmers get paid more, apparently the people programming in Rust really like Rust, more so than say, Python programmers loved Python - so that's good to know, but it's that and not anything else.
  - wiseowise 2y ago
    Tiobe is garbage. I remember Python making waves since 2005 with Google using it and such.
    - Numerlor 2y ago
      From what I can tell it wasn't as prominent as it has been recently, with being a popular pick for random projects that weren't just gluing things together. The big companies that used it were perfectly happy specializing the interpreter to their use case instead of upstreaming general improvements
  - collinmanderson 2y ago
    python had momentum until the 2->3 transition put a huge damper on it around 2012-2016. Python got lucky with the machine learning community using Python. (Thank you TensorFlow and PyTorch, and the SciPy community for saving Python.)
woodruffw 2y ago
Neither Java nor JS encourages the use of native extensions to the same degree that Python does. So some of it is a fundamental difference in approach: Python has gotten very far by offloading hot paths into native code instead of optimizing the interpreter itself. (Recent positive developments in Python’s interpreted performance have subverted this informal tendency.)
- throwup238 2y ago
  Node also introduced a stable extension API that people could build native code against relatively early in its history compared to Python. That and the general velocity of the V8 interpreter and its complex API kept developers from reaching in like they did with Python, or leaving tons of libraries in the ecosystem that are too critical to drop.
  - woodruffw 2y ago
    Yeah, I think it's mostly about complexity: CPython's APIs also change quite a bit, but they're pretty simple (in the "simple enough to hang yourself with" sense).
- sieve 2y ago
  > Neither Java nor JS encourages the use of native extensions to the same degree that Python does. You already had billions of lines of Java and JS code that HAD to be sped up. So they had no alternative. If python had gone down the same route, speeding it up without caveats would have been that much easier.
  - woodruffw 2y ago
    I don't think that's the reason. All three ecosystems had the same inflection point, and chose different solutions to it. Python's was especially "easy" since the C API was already widely used and there were no other particular constraints (WORA for Java, pervasive async for JS) that impeded it.
zerkten 2y ago
>> My use of python is somewhat recent. But the two languages that I have used a lot of - Java and JS - have interpreters that were heavily optimized over time. I wonder why that never happened with python and, instead, everyone continues to write their critical code in C/Rust. Improving Python performance has been a topic as far back as 2008 when I attended my first PyCon. A quick detour on Python 3 because there is some historical revisionism because many online people weren't around in the earlier days. Back then the big migration to Python 3 was in front of the community. The timeline concerns that popped up when Python really got steam in the industry between 2012 and 2015 weren't as huge a concern. You can refer to Guido's talks from PyCon 2008 and 2009 if they are available somewhere to get the vibe on the urgency. Python is impactful because it changes the language and platform while requiring a massive amount of effort. Back to perf. Around 2008, there was a feeling that an alternative to CPython might be the future. Candidates included IronPython, Jython, and PyPy. Others like Unladen Swallow wanted to make major changes to CPython (https://peps.python.org/pep-3146/ https://peps.python.org/pep-3146/). Removing the GIL was another direction people wanted to take because it seemed simpler in a way. This is a well researched area with David Beazley having many talks like this oldie (https://www.youtube.com/watch?v=ph374fJqFPE https://www.youtube.com/watch?v=ph374fJqFPE). The idea is much older (https://dabeaz.blogspot.com/2011/08/inside-look-at-gil-removal-patch-of.html https://dabeaz.blogspot.com/2011/08/inside-look-at-gil-remov...). All of these alternative implementations of Python from this time period have basically failed at the goal of replacing CPython. IronPython was a Python 2 implmentation and updating to Python 3 while trying grow to challenge CPython was impossible. Eventually, Microsoft lost interest and that was that. Similar things happened for the others. GIL removal was a constant topic from 2008 until recently. Compatibility of extensions was a major concern causing inertia and the popularity meant even more C/C++/Rust code relying on a GIL. The option to disable (https://peps.python.org/pep-0703/ https://peps.python.org/pep-0703/) only happened because the groundwork was eventually done properly to help the community move. The JVM has very clearly defined interfaces and specs similar to the CLR which make optimization viable. JS doesn't have the compatibility concerns. That was just a rough overview but many of the stories of Python woes miss a lot of this context. Many discussions about perf over the years have descended into a GIL discussion without any data to show the GIL would change performance. People love to talk about it but turn out to be IO-bound when you profile code.
physicsguy 2y ago
For scientific stuff and ML, it's because people already had libraries written in C/Fortran/C++ and so calling it directly just made sense. In other languages that didn't happen and you don't have anywhere near as good scientific/ML packages as a result.

by sieve 2y ago

Numerlor 2y ago

python just didn't have much momentum until relatively recently, despite it's age. There are efforts to speed it up going on now backed by Microsoft. For pypy it's in a weird spot as the things it does fast are the ones you'd usually just offload to a module implemented in C

sieve 2y ago
> There are efforts to speed it up Well, the extensions are going to complicate this a lot. A fast JIT interpreter cannot reach into a library and do its magic there the way HotSpot/V8 can with native Java/JS code.
simonw 2y ago
As a long time Pythonista I was going to push back against your suggestion that Python didn't have much momentum until recently, but then I looked at the historic graph on https://www.tiobe.com/tiobe-index/ https://www.tiobe.com/tiobe-index/ and yeah, Python's current huge rise in popularity didn't really get started until around 2018. (TIOBE's methodology is a bit questionable though, as far as I can tell it's almost entirely based on how many search engine hits they get for "X programming". https://www.tiobe.com/tiobe-index/programminglanguages_definition/ https://www.tiobe.com/tiobe-index/programminglanguages_defin...)
- tialaramex 2y ago
  Yes, TIOBE is garbage. The biggest problem is that because they're coy about methodology we don't even know what we're talking about. Rust's "Most Loved" Stack Overflow numbers were at least a specific thing where you can say OK that doesn't mean there's more Rust software or that Rust programmers get paid more, apparently the people programming in Rust really like Rust, more so than say, Python programmers loved Python - so that's good to know, but it's that and not anything else.
- wiseowise 2y ago
  Tiobe is garbage. I remember Python making waves since 2005 with Google using it and such.
  - Numerlor 2y ago
    From what I can tell it wasn't as prominent as it has been recently, with being a popular pick for random projects that weren't just gluing things together. The big companies that used it were perfectly happy specializing the interpreter to their use case instead of upstreaming general improvements
- collinmanderson 2y ago
  python had momentum until the 2->3 transition put a huge damper on it around 2012-2016. Python got lucky with the machine learning community using Python. (Thank you TensorFlow and PyTorch, and the SciPy community for saving Python.)

woodruffw 2y ago

Neither Java nor JS encourages the use of native extensions to the same degree that Python does. So some of it is a fundamental difference in approach: Python has gotten very far by offloading hot paths into native code instead of optimizing the interpreter itself. (Recent positive developments in Python’s interpreted performance have subverted this informal tendency.)

throwup238 2y ago
Node also introduced a stable extension API that people could build native code against relatively early in its history compared to Python. That and the general velocity of the V8 interpreter and its complex API kept developers from reaching in like they did with Python, or leaving tons of libraries in the ecosystem that are too critical to drop.
- woodruffw 2y ago
  Yeah, I think it's mostly about complexity: CPython's APIs also change quite a bit, but they're pretty simple (in the "simple enough to hang yourself with" sense).
sieve 2y ago
> Neither Java nor JS encourages the use of native extensions to the same degree that Python does. You already had billions of lines of Java and JS code that HAD to be sped up. So they had no alternative. If python had gone down the same route, speeding it up without caveats would have been that much easier.
- woodruffw 2y ago
  I don't think that's the reason. All three ecosystems had the same inflection point, and chose different solutions to it. Python's was especially "easy" since the C API was already widely used and there were no other particular constraints (WORA for Java, pervasive async for JS) that impeded it.

zerkten 2y ago

>> My use of python is somewhat recent. But the two languages that I have used a lot of - Java and JS - have interpreters that were heavily optimized over time. I wonder why that never happened with python and, instead, everyone continues to write their critical code in C/Rust. Improving Python performance has been a topic as far back as 2008 when I attended my first PyCon. A quick detour on Python 3 because there is some historical revisionism because many online people weren't around in the earlier days. Back then the big migration to Python 3 was in front of the community. The timeline concerns that popped up when Python really got steam in the industry between 2012 and 2015 weren't as huge a concern. You can refer to Guido's talks from PyCon 2008 and 2009 if they are available somewhere to get the vibe on the urgency. Python is impactful because it changes the language and platform while requiring a massive amount of effort. Back to perf. Around 2008, there was a feeling that an alternative to CPython might be the future. Candidates included IronPython, Jython, and PyPy. Others like Unladen Swallow wanted to make major changes to CPython (https://peps.python.org/pep-3146/ https://peps.python.org/pep-3146/). Removing the GIL was another direction people wanted to take because it seemed simpler in a way. This is a well researched area with David Beazley having many talks like this oldie (https://www.youtube.com/watch?v=ph374fJqFPE https://www.youtube.com/watch?v=ph374fJqFPE). The idea is much older (https://dabeaz.blogspot.com/2011/08/inside-look-at-gil-removal-patch-of.html https://dabeaz.blogspot.com/2011/08/inside-look-at-gil-remov...). All of these alternative implementations of Python from this time period have basically failed at the goal of replacing CPython. IronPython was a Python 2 implmentation and updating to Python 3 while trying grow to challenge CPython was impossible. Eventually, Microsoft lost interest and that was that. Similar things happened for the others. GIL removal was a constant topic from 2008 until recently. Compatibility of extensions was a major concern causing inertia and the popularity meant even more C/C++/Rust code relying on a GIL. The option to disable (https://peps.python.org/pep-0703/ https://peps.python.org/pep-0703/) only happened because the groundwork was eventually done properly to help the community move. The JVM has very clearly defined interfaces and specs similar to the CLR which make optimization viable. JS doesn't have the compatibility concerns. That was just a rough overview but many of the stories of Python woes miss a lot of this context. Many discussions about perf over the years have descended into a GIL discussion without any data to show the GIL would change performance. People love to talk about it but turn out to be IO-bound when you profile code.

physicsguy 2y ago

For scientific stuff and ML, it's because people already had libraries written in C/Fortran/C++ and so calling it directly just made sense. In other languages that didn't happen and you don't have anywhere near as good scientific/ML packages as a result.