3 ms·

There are faster Python implementations, namely Pypy. I think one of the reasons Pypy isn't more popular is because it's C interoperability story is weaker than

by weberc2 10y ago

There are faster Python implementations, namely Pypy. I think one of the reasons Pypy isn't more popular is because it's C interoperability story is weaker than the reference Python implementation (CPython--not to be confused with Cython), or at least it's not compatible with many important C-based Python libraries (e.g., Pandas).

adrianratnapala 10y ago
The C-compatbility story works both ways. I usually roll my eyes when people talk about making dynamic lanuages fast -- what's the point? In those cases where speed matters - use static language. As a result there is little incentive to use something like PyPy (although PyPy does have other nice features in addition to raw speed). But on the web there is a point. Unless you can use HTML alone, you have to use JavaScript -- and you can't just plug in a module that you wrote in C.
- cmdrfred 10y ago
  Can't you? Maybe not today, but soon. https://webassembly.github.io/ https://webassembly.github.io/
- static_noise 10y ago
  It's about managing your resources. You can simply buy the fastest processor for cheap money. You can buy systems with many cores or even build a cluster for considerable money. But you can not buy time. Developing python applications in many cases is much quicker than developing comparable applications in C++, C or Assembler. Therefore it makes a lot of sense for many companies to choose tools with quick development time to get the product running and care for performance later. Since there may be so much code be written in python, it cannot simply be converted into a faster language since most critical parts are already accelerated using the proper libraries provided by the language ecosystem. Porting a complex application to another language in many cases is a really hard problem and might even not be possible to be done in smaller steps. Therefore we need every bit of performance we can squeeze out of python. The whole world will gain from that. If we manage great strides like JavaScript it would be glorious!
  - cmdrfred 10y ago
    Also soon enough (if we aren't there already), fast hardware will be so cheap that the speed of execution at least for end user applications will rarely be a factor. Sure that database application you wrote in C is measurably faster, but can a human being detect the difference? More and more the response will be no.
    - weberc2 10y ago
      In the last 5 years, clock speeds have stagnated. Hardware innovation has revolved around increasing cores and improving power efficiency. Unfortunately, Python and most other dynamic languages are poorly suited to take advantage of parallelism, so your C database can only get faster, but your Python database will not.
      - pcwalton 10y ago
        Honestly, the vast majority of languages are bad at parallelism. Dynamic vs. static doesn't have much to do with it. Task-parallel multicore (like goroutines and Web Workers and the like give you) is only a small fraction of what it means to "take advantage of parallelism", and if you're only using that (as opposed to SIMD, GPUs, concurrent data structures, work-stealing, etc.) then you are likely to see disappointing speedups in practice. This is especially true in the last 5 years, which contrary to your post has seen a shift away from multicore task-parallel hardware (core counts are not increasing) and toward data-parallel hardware (wider SIMD lanes, better on-board GPUs, APU architectures, etc.)
        weberc2 10y ago
        You're missing the point. There's "suboptimal parallelism", and then there's "no parallelism". Python and many other dynamic languages fall into the latter category.
        BuckRogers 10y ago
        I'd call Python optimal parallelism. I prefer the scaling by instance method of Node and Python over the old ways- writing multithreaded Java apps. People can knock themselves out writing stuff like that if they want to. But my purposes have been well-served scaling by spinning up instances per-core. For me, explicit pthreading should never be done except with systems software. In which case you're going to be using a systems language like C, C++ or Rust anyway. Not the oddball middletier stuff that isn't high or low level like Go, Java and C#.
- weberc2 10y ago
  By that logic, why optimize anything? Just use the thing that's out there that's already faster. Clearly some people think there's a benefit to dynamic languages (I'm not one of them), and they'd like to preserve that benefit while minimizing the performance impact.
- Animats 10y ago
  Python connecting to C is used mostly to attach to large C objects that do something complicated, like OpenSSL. Math inner loop intensive things like numpy could potentially be optimized up to C speed by PyPy. (At least if you could create an array of a fixed type in Python.)
Animats 10y ago
"I think one of the reasons Pypy isn't more popular is because its C interoperability story is weaker than the reference Python implementation" The CPython crowd apparently made some change between CPython 3.4 and CPython 3.5 which changed how Windows DLLs are called. This broke Py2exe, which no longer works for the current release of CPython. [1] CPython's approach to C extensions usually requires having the same Microsoft C compiler version used to build Python. This is now a problem for Python 2.7, because the Microsoft compiler version used to build it is no longer available. Microsoft recently changed their API for the C runtime, and CPython has been changed accordingly. CPython's "C interoperability story" isn't really that good on Windows. [1] http://stevedower.id.au/blog/building-for-python-3-5/ http://stevedower.id.au/blog/building-for-python-3-5/
- enos 10y ago
  This is true on other OSes as well. C extensions are assumed to be compiled with the same compiler as Python itself. If you use distutils it will also use the same flags. This is less problematic on other OSes where GCC dominates and clang is compatible, but it can bite on HPC systems that use PGI compilers.
- int_19h 10y ago
  It's good enough for practical purposes - sure, you have to build a separate version of your package for every different version of VC++ that is used by your supported versions of Python, but we're still talking about 3-4 binaries top. When that is not enough, you can still recompile manually if need be, so long as source for the module is available (which it is in most cases). On the other hand, native modules are such a big part of the Python ecosystem, that not supporting the API and ABI at all confines any alternative implementations to a very small niche.
- ihnorton 10y ago
  > This is now a problem for Python 2.7, because the Microsoft compiler version used to build it is no longer available. There is a CMake build system for 2.7 that works great with VS2013-2015: https://github.com/python-cmake-buildsystem/python-cmake-buildsystem https://github.com/python-cmake-buildsystem/python-cmake-bui...
int_19h 10y ago
Have a look at this: https://github.com/Microsoft/Pyjion https://github.com/Microsoft/Pyjion

by weberc2 10y ago

adrianratnapala 10y ago

The C-compatbility story works both ways. I usually roll my eyes when people talk about making dynamic lanuages fast -- what's the point? In those cases where speed matters - use static language. As a result there is little incentive to use something like PyPy (although PyPy does have other nice features in addition to raw speed). But on the web there is a point. Unless you can use HTML alone, you have to use JavaScript -- and you can't just plug in a module that you wrote in C.

cmdrfred 10y ago
Can't you? Maybe not today, but soon. https://webassembly.github.io/ https://webassembly.github.io/
static_noise 10y ago
It's about managing your resources. You can simply buy the fastest processor for cheap money. You can buy systems with many cores or even build a cluster for considerable money. But you can not buy time. Developing python applications in many cases is much quicker than developing comparable applications in C++, C or Assembler. Therefore it makes a lot of sense for many companies to choose tools with quick development time to get the product running and care for performance later. Since there may be so much code be written in python, it cannot simply be converted into a faster language since most critical parts are already accelerated using the proper libraries provided by the language ecosystem. Porting a complex application to another language in many cases is a really hard problem and might even not be possible to be done in smaller steps. Therefore we need every bit of performance we can squeeze out of python. The whole world will gain from that. If we manage great strides like JavaScript it would be glorious!
- cmdrfred 10y ago
  Also soon enough (if we aren't there already), fast hardware will be so cheap that the speed of execution at least for end user applications will rarely be a factor. Sure that database application you wrote in C is measurably faster, but can a human being detect the difference? More and more the response will be no.
  - weberc2 10y ago
    In the last 5 years, clock speeds have stagnated. Hardware innovation has revolved around increasing cores and improving power efficiency. Unfortunately, Python and most other dynamic languages are poorly suited to take advantage of parallelism, so your C database can only get faster, but your Python database will not.
    - pcwalton 10y ago
      Honestly, the vast majority of languages are bad at parallelism. Dynamic vs. static doesn't have much to do with it. Task-parallel multicore (like goroutines and Web Workers and the like give you) is only a small fraction of what it means to "take advantage of parallelism", and if you're only using that (as opposed to SIMD, GPUs, concurrent data structures, work-stealing, etc.) then you are likely to see disappointing speedups in practice. This is especially true in the last 5 years, which contrary to your post has seen a shift away from multicore task-parallel hardware (core counts are not increasing) and toward data-parallel hardware (wider SIMD lanes, better on-board GPUs, APU architectures, etc.)
      - weberc2 10y ago
        You're missing the point. There's "suboptimal parallelism", and then there's "no parallelism". Python and many other dynamic languages fall into the latter category.
        BuckRogers 10y ago
        I'd call Python optimal parallelism. I prefer the scaling by instance method of Node and Python over the old ways- writing multithreaded Java apps. People can knock themselves out writing stuff like that if they want to. But my purposes have been well-served scaling by spinning up instances per-core. For me, explicit pthreading should never be done except with systems software. In which case you're going to be using a systems language like C, C++ or Rust anyway. Not the oddball middletier stuff that isn't high or low level like Go, Java and C#.
weberc2 10y ago
By that logic, why optimize anything? Just use the thing that's out there that's already faster. Clearly some people think there's a benefit to dynamic languages (I'm not one of them), and they'd like to preserve that benefit while minimizing the performance impact.
Animats 10y ago
Python connecting to C is used mostly to attach to large C objects that do something complicated, like OpenSSL. Math inner loop intensive things like numpy could potentially be optimized up to C speed by PyPy. (At least if you could create an array of a fixed type in Python.)

Animats 10y ago

"I think one of the reasons Pypy isn't more popular is because its C interoperability story is weaker than the reference Python implementation" The CPython crowd apparently made some change between CPython 3.4 and CPython 3.5 which changed how Windows DLLs are called. This broke Py2exe, which no longer works for the current release of CPython. [1] CPython's approach to C extensions usually requires having the same Microsoft C compiler version used to build Python. This is now a problem for Python 2.7, because the Microsoft compiler version used to build it is no longer available. Microsoft recently changed their API for the C runtime, and CPython has been changed accordingly. CPython's "C interoperability story" isn't really that good on Windows. [1] http://stevedower.id.au/blog/building-for-python-3-5/ http://stevedower.id.au/blog/building-for-python-3-5/

enos 10y ago
This is true on other OSes as well. C extensions are assumed to be compiled with the same compiler as Python itself. If you use distutils it will also use the same flags. This is less problematic on other OSes where GCC dominates and clang is compatible, but it can bite on HPC systems that use PGI compilers.
int_19h 10y ago
It's good enough for practical purposes - sure, you have to build a separate version of your package for every different version of VC++ that is used by your supported versions of Python, but we're still talking about 3-4 binaries top. When that is not enough, you can still recompile manually if need be, so long as source for the module is available (which it is in most cases). On the other hand, native modules are such a big part of the Python ecosystem, that not supporting the API and ABI at all confines any alternative implementations to a very small niche.
ihnorton 10y ago
> This is now a problem for Python 2.7, because the Microsoft compiler version used to build it is no longer available. There is a CMake build system for 2.7 that works great with VS2013-2015: https://github.com/python-cmake-buildsystem/python-cmake-buildsystem https://github.com/python-cmake-buildsystem/python-cmake-bui...

int_19h 10y ago

Have a look at this: https://github.com/Microsoft/Pyjion https://github.com/Microsoft/Pyjion