3 ms·

As someone in the Julia ecosystem, I wouldn't know where to begin with writing performant python. Is it Numba? Taichi? Torchscript? NumPy? Cython? Pypy? None of

As someone in the Julia ecosystem, I wouldn't know where to begin with writing performant python. Is it Numba? Taichi? Torchscript? NumPy? Cython? Pypy? None of these seem to work together (besides calling each other, I suppose).

montebicyclelo 4y ago
Wow, it seems like you have a number of good options to choose from. What a horrible situation, to have a language with a really rich ecosystem of powerful libraries.
- adgjlsfhk1 4y ago
  The problem is you have 10 different options that all require rewriting your code and you can't share code with anyone who used any of the others (and almost all of them are still pretty slow compared to C++/Julia)
- jeron 4y ago
  decision paralysis is a real thing
- switchbak 4y ago
  10 options to make up for the deficiencies of the base language? I wouldn't call that a good thing.
  - xiaodai 4y ago
    amen
- jakobnissen 4y ago
  In my view they are a broad selection of bad choices, neither of whom will generally work to write fast code. I tried them all on my last large Python project (admittedly, 5 years ago since I switched to Julia since). Want to use NumPy? Better hope you have code that can be expressed as array computations (a tiny minority of performance sensitive tasks in my field of work). Cython? Now you're just writing another language, and get the joy of having to distribute compiled code in your Python package - for example, like the very fun segfaults I experienced because Conda will automatically override the system linker, breaking all compilation including my Cython modules. Numba? Hope you don't make custom classes - you know, one of the basic features of the language.
ActorNightly 4y ago
Performant code is really a big group of code. If you look at ML, Python is completely fine because all the processing that happens with matrix multiplication, even on CPUs, far, far, FAR outweighs all the setup stuff in volume of operations. On the other hand, if majority of your application relies heavily on processing speed (i.e you need compare/jump operations rather just add/multiply/load/store of the GPUs), Python is going to be slow. In this case, if you want custom performant code, you write C extensions for the performant critical code, and launch them from higher level python code. That being said, there is generally a library (like Taichi) that already does this for you.

montebicyclelo 4y ago

Wow, it seems like you have a number of good options to choose from. What a horrible situation, to have a language with a really rich ecosystem of powerful libraries.

adgjlsfhk1 4y ago
The problem is you have 10 different options that all require rewriting your code and you can't share code with anyone who used any of the others (and almost all of them are still pretty slow compared to C++/Julia)
jeron 4y ago
decision paralysis is a real thing
switchbak 4y ago
10 options to make up for the deficiencies of the base language? I wouldn't call that a good thing.
- xiaodai 4y ago
  amen
jakobnissen 4y ago
In my view they are a broad selection of bad choices, neither of whom will generally work to write fast code. I tried them all on my last large Python project (admittedly, 5 years ago since I switched to Julia since). Want to use NumPy? Better hope you have code that can be expressed as array computations (a tiny minority of performance sensitive tasks in my field of work). Cython? Now you're just writing another language, and get the joy of having to distribute compiled code in your Python package - for example, like the very fun segfaults I experienced because Conda will automatically override the system linker, breaking all compilation including my Cython modules. Numba? Hope you don't make custom classes - you know, one of the basic features of the language.

ActorNightly 4y ago

Performant code is really a big group of code. If you look at ML, Python is completely fine because all the processing that happens with matrix multiplication, even on CPUs, far, far, FAR outweighs all the setup stuff in volume of operations. On the other hand, if majority of your application relies heavily on processing speed (i.e you need compare/jump operations rather just add/multiply/load/store of the GPUs), Python is going to be slow. In this case, if you want custom performant code, you write C extensions for the performant critical code, and launch them from higher level python code. That being said, there is generally a library (like Taichi) that already does this for you.