3 ms·

He, Inada Naoki and Victor Stinner on the optimization front are some developers that are capable of pulling a JIT of CPython. This is something that I feel th

by bratao 6y ago

He, Inada Naoki and Victor Stinner on the optimization front are some developers that are capable of pulling a JIT of CPython. This is something that I feel that Python need to do to keep the relevance. .Net Core, Rust and GoLang are examples of modern languages that deliver on the performance front. I love Python and it is what use everyday. But very often I feel the need to a better code generation. Numba and Cython feels like a glue. PyPy it is slow with extensions, and this is a huge part of Python ecosystem. Although HPy (https://github.com/hpyproject/hpy https://github.com/hpyproject/hpy) is trying to change this. I feel that this is a must to Python, and would love to contribute money to this task.

oefrha 6y ago
To extend what you said a bit: PyPy is not only slow with extensions, it's also massively slower for running short-lived code that isn't repeated much, which is precisely the scenario of many if not most command line tools written in Python. Although I'm not sure there are enough people concerned about Python's performance in this scenario. And yeah, I would be happy to pitch in too if CPython can be 50% faster reliably. It only takes 500 people each pitching in $100 to reach $50k. Edit: somehow got $50k etched in mind when it says $500k. That’s more difficult from individual funding.
- dr_zoidberg 6y ago
  OTOH, $2M is a sneeze for large corporations. I totally understand why they want to push community funding, but I wonder if they're open to having corporate sponsors or something along those lines. The idea would be "yes, you pay the bill, but you don't get to push an agenda with this".
throwaway894345 6y ago
It’s a chicken and egg situation. If Python weren’t so slow, there wouldn’t be such a heavy reliance on C extensions in the ecosystem, and consequently it would be easier to optimize CPython and PyPy because there would be less at stake with respect to breaking the ecosystem. Not only that, but lots of other things get better if C-extensions go away: much less need for executable project files which dramatically improves package manager performance (no need to recursively download a package and run setup.py to determine the next level of dependencies just to build the dependency tree), packaging gets dramatically easier (C projects have no standard reproducible build system so every C project including C extension projects must have its own package recipe and it’s often impractical to support more than a handful of target systems), etc. Hpy appears to be the right approach to fixing this, but there needs to be a concerted effort to migrate the ecosystem toward it and then deprecate the old expansive interface. And then at some point in the distant future we can expect things to be as nice as other ecosystems are today.
- throwaway189262 6y ago
  This is absolutely the problem. Fast languages like C#, Java, Go and even JS almost nothing is native. It's normal to have even large projects with no FFI'd native code. Python and Ruby are trapped in the C extensions spiral. Languages are slow so everything uses C bindings, which makes it hard to speed up the language
  - pjmlp 6y ago
    And in what concerns Java and .NET platforms, it has been acknowledged that too much of low level capabilities have been left out and those features are being added, even if slower than many of us would like to, so eventually even the stuff that might still require a native library today, won't be necessary in the future.
- japanuspus 6y ago
  Absolutely agree. I do believe this will be hard to change, though: Coming from academia my first Python experience was as a front-end to actively developed Fortran code. In that segment, the easy link to external code is one of the main selling points of Python.
- murgindrag 6y ago
  Translation layer. There ought to be a thick, fat, slow layer between Fast Python and CPython. You can write fast native you. You can write fast code by calling into native. The two rarely cross. If we were clever, we could even have one front-end and multiple backends to go into fast Python, CPython, Cython, embedded micropython, etc. If we were double-plus-clever, we'd add GPGPU, multicore, and compute cluster to the mix. We'd need someone with deep pockets and high risk tolerance to pull it off. It's a big change. Big changes often fail. Still, when I look at the amount of Python at Google... Well, it'd sure be a pity if all of that became legacy code when we all jumped ship to Julia.
quotemstr 6y ago
Thanks for the link to Hpy. That's a fantastic project

by bratao 6y ago

oefrha 6y ago

To extend what you said a bit: PyPy is not only slow with extensions, it's also massively slower for running short-lived code that isn't repeated much, which is precisely the scenario of many if not most command line tools written in Python. Although I'm not sure there are enough people concerned about Python's performance in this scenario. And yeah, I would be happy to pitch in too if CPython can be 50% faster reliably. It only takes 500 people each pitching in $100 to reach $50k. Edit: somehow got $50k etched in mind when it says $500k. That’s more difficult from individual funding.

dr_zoidberg 6y ago
OTOH, $2M is a sneeze for large corporations. I totally understand why they want to push community funding, but I wonder if they're open to having corporate sponsors or something along those lines. The idea would be "yes, you pay the bill, but you don't get to push an agenda with this".

throwaway894345 6y ago

It’s a chicken and egg situation. If Python weren’t so slow, there wouldn’t be such a heavy reliance on C extensions in the ecosystem, and consequently it would be easier to optimize CPython and PyPy because there would be less at stake with respect to breaking the ecosystem. Not only that, but lots of other things get better if C-extensions go away: much less need for executable project files which dramatically improves package manager performance (no need to recursively download a package and run setup.py to determine the next level of dependencies just to build the dependency tree), packaging gets dramatically easier (C projects have no standard reproducible build system so every C project including C extension projects must have its own package recipe and it’s often impractical to support more than a handful of target systems), etc. Hpy appears to be the right approach to fixing this, but there needs to be a concerted effort to migrate the ecosystem toward it and then deprecate the old expansive interface. And then at some point in the distant future we can expect things to be as nice as other ecosystems are today.

throwaway189262 6y ago
This is absolutely the problem. Fast languages like C#, Java, Go and even JS almost nothing is native. It's normal to have even large projects with no FFI'd native code. Python and Ruby are trapped in the C extensions spiral. Languages are slow so everything uses C bindings, which makes it hard to speed up the language
- pjmlp 6y ago
  And in what concerns Java and .NET platforms, it has been acknowledged that too much of low level capabilities have been left out and those features are being added, even if slower than many of us would like to, so eventually even the stuff that might still require a native library today, won't be necessary in the future.
japanuspus 6y ago
Absolutely agree. I do believe this will be hard to change, though: Coming from academia my first Python experience was as a front-end to actively developed Fortran code. In that segment, the easy link to external code is one of the main selling points of Python.
murgindrag 6y ago
Translation layer. There ought to be a thick, fat, slow layer between Fast Python and CPython. You can write fast native you. You can write fast code by calling into native. The two rarely cross. If we were clever, we could even have one front-end and multiple backends to go into fast Python, CPython, Cython, embedded micropython, etc. If we were double-plus-clever, we'd add GPGPU, multicore, and compute cluster to the mix. We'd need someone with deep pockets and high risk tolerance to pull it off. It's a big change. Big changes often fail. Still, when I look at the amount of Python at Google... Well, it'd sure be a pity if all of that became legacy code when we all jumped ship to Julia.

quotemstr 6y ago

Thanks for the link to Hpy. That's a fantastic project