4 ms·

Just for reference, * Nuitka[0] "is a Python compiler written in Python. It's fully compatible with Python 2.6, 2.7, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, 3.10, and 3.

by fernly 3y ago

Just for reference, * Nuitka[0] "is a Python compiler written in Python. It's fully compatible with Python 2.6, 2.7, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, 3.10, and 3.11." * Pypy[1] "is a replacement for CPython" with builtin optimizations such as on the fly JIT compiles. * Cython[2] "is an optimising static compiler for both the Python programming language and the extended Cython programming language... makes writing C extensions for Python as easy as Python itself." * Numba[3] "is an open source JIT compiler that translates a subset of Python and NumPy code into fast machine code." * Pyston[4] "is a performance-optimizing JIT for Python, and is drop-in compatible with ... CPython 3.8.12" [0] https://github.com/Nuitka/Nuitka https://github.com/Nuitka/Nuitka [1] https://www.pypy.org/ https://www.pypy.org/ [2] https://cython.org/ https://cython.org/ [3] https://numba.pydata.org/ https://numba.pydata.org/ [4] https://github.com/pyston/pyston https://github.com/pyston/pyston

Scarbutt 3y ago
And Cinder
dragonwriter 3y ago
While we’re at it: * Mypyc [0] “compiles Python modules to C extensions. It uses standard Python type hints to generate fast code. Mypyc uses mypy to perform type checking and type inference. Mypyc can compile anything from one module to an entire codebase. The mypy project has been using mypyc to compile mypy since 2019, giving it a 4x performance boost over regular Python.” [0] https://github.com/mypyc/mypyc https://github.com/mypyc/mypyc
- derbOac 3y ago
  ... and also now there's Mojo too (https://www.modular.com/mojo https://www.modular.com/mojo). Not the same but seems relevant.
  - maegul 3y ago
    Yea it’s gotten to the point where it’s a sub-domain of expertise in Python being on top of the various performance extensions/runtimes/hacks. Intuitively, numba and cython still feel like the most interesting/relevant. I recall talk a few years ago about the possibility of writing packages in “numba”. IE, targeting the subset of Python and numpy that numba supports. Occasionally I check in with numba’s releases and they seem to be slowly but surely supporting more of Python such that surely it will start to make sense to talk about the “numba” language. Has anyone got a tighter grip on whether this than I?
    - BiteCode_dev 3y ago
      I would say the most popular way to speed up a python program today is not through a compiler anymore, but using rust for hot loops and maturin (https://pypi.org/project/maturin/ https://pypi.org/project/maturin/) for seamlessly use it in Python and provide packages. I get the benefit of being able to "just use python" and still gain a speed up though. Plus there are many situations with rust is no possible (no time to learn such a complex language, rust is not approved by the security team).
      - tialaramex 3y ago
        The other edge is where your heatmap is just all red, and so "speed up hot loops" isn't going to get it done, and at that point somebody needs to bite the bullet and {hire people who know / learn} an AOT language with good performance. or it turns out the reason the heatmap is all red is that you were solving categorically the wrong problem, e.g. you decided to use machine vision to figure out what's in the packaging, while your competitors are just scanning the EAN-13 barcode.
  - BiteCode_dev 3y ago
    Careful though, while Mojo is exciting, it's very tied to the modular platform for now, unlike the other projects.
  - gkbrk 3y ago
    All the other options that were mentioned are open-source and available to public usage. They actually exist. Mojo on the other hand has an empty Github repo, no downloads, and no public access. You need to get approval from Modular Inc before you can even use a demo of Mojo. Clicking the "Get started with Mojo" or the "Request access" buttons take you to the same page with a form that asks your full name and email address, but giving them your personal info does not actually give you access to Mojo. It just gives you the following message until someone at Modular Inc vets your application. > We will contact you as soon as we are ready to onboard you into our early preview program. All the other options mentioned in this thread actually have downloads and repos that you can check out, and you don't need to fill an application form to "Request access". They exist, right now.
    - derbOac 3y ago
      Yeah I didn't mean to advocate for Mojo necessarily. I use Python for projects and like a lot about it, but often feel as if its implementations are kind of fragmented in many respects, and Mojo seems like it could potentially further that trend for me at least, at the current time. Other projects are certainly more open and established. For me personally what I want to see is a completely open, working, high-performance Python compiler that is fully compatible with official spec 100%, in the sense that you could take any code anywhere and it would work just as well on the official interpreter as on the compiler. Maybe this means creating an official Python 4.0 spec or something that's a superset of 3.0, or something, but right now I'm not sure I see anything quite like that. I could see Mojo being that eventually but you're right that at the moment it's far from that.
bloaf 3y ago
While we're at it: * Taichi "is embedded in Python and uses modern just-in-time (JIT) frameworks (for example LLVM, SPIR-V) to offload the Python source code to native GPU or CPU instructions, offering the performance at both development time and runtime." https://www.taichi-lang.org/ https://www.taichi-lang.org/
- longqzh 3y ago
  Very interested in the benchmark between Taichi and numba, both of them are using llvm as backend
prpl 3y ago
And Jax, and now Mojo, and while we’re at it, Graal.Python, and all sorts of other forgotten ones.
nologic01 3y ago
A wikipedia entry with a side by side comparison would seem appropriate at this stage
nurettin 3y ago
Cython lets you incrementally add types to your python code and transpile to C. Its best feature is the html output that highlights which part of your code still has an external dependency and is thus not directly translatable to C. Numba has an amazing jit decorator. It can even build on numpy functions. Once I had a simple sum loop and it was compiled into O(1). Pyston is very cool, it outputs C++ code that you can statically build into your project. Still needs libpython. Pypy speeds up plain python code. It can even install from pip. I tried it on neat-python package, solid 40% improvement. Nuitka is different. It is a packaging system which embeds the python interpreter and your code into a distributable binary.
helsinkiandrew 3y ago
Two questions: 1) Why so many Python compilers - should python be more open/modular so different JITs and other functionality can be plugged in/chosen at install or runtime? 2) Why has no one named a compiler "Monty"?
- BiteCode_dev 3y ago
  They don't have the same goals, nor characteristics. Numba and cython are rarely used to write the entire program, you usually apply them on your bottleneck, and compile just this part. Also JIT, like for numba or pypy, compile when the program runs, but projects like cython and nuikta compile ahead of time. The performance characteristics are very different, since the former need a warmup but the latter don't, however they need to be manually built. Solutions like cython or mypyc require type hinting for getting the best speed up. All of them except nuikta target better perfs, some in general (pyston, mypyc, cython), some targetting specifically numerical calculations (numba). nuitka doesn't care about any of that, it takes regular Python code and compiles it. While it does provide some speedup (up to 4x), it's not the goal at all. The goal of nuikta is to able you to ship your python program as a standalone executable to the end user. It's the most reliable way to do it, as, just like the tag line says, it's "extremly compatible". I've yet to find a Python program that nuikta couldn't compile, including one with PyQT + numpy. It's very good at it.
t-vi 3y ago
Not to forget CPython's own faster-cpython project which aims at JIT compiling. [0] Also JAX[1], PyTorch[2] come with JIT compilation specifically aimed at GPU kernels "fusing" multiple higher-level operation And NumPy/Scipy (also) uses Pythran[3], an AOT compiler not too unsimilar to Numba. [0] https://github.com/faster-cpython/cpython https://github.com/faster-cpython/cpython [1] https://pytorch.org/docs/stable/generated/torch.compile.html https://pytorch.org/docs/stable/generated/torch.compile.html [2] https://jax.readthedocs.io/ https://jax.readthedocs.io/ [3] https://pythran.readthedocs.io/ https://pythran.readthedocs.io/ I think some useful classification criteria would be - does it replace running code in Python (either own interpreter or compiler), vs does it speed up certain bits, - does it aim to faithfully implement Python or does it intentionally diverge in the semantics, - does it provide low-level semantics (where numba, pythran shine) or higher-level (e.g. what PyTorch, JAX do) - target architectures (CPU, GPU offloading, ...)

by fernly 3y ago

Scarbutt 3y ago

And Cinder

dragonwriter 3y ago

While we’re at it: * Mypyc [0] “compiles Python modules to C extensions. It uses standard Python type hints to generate fast code. Mypyc uses mypy to perform type checking and type inference. Mypyc can compile anything from one module to an entire codebase. The mypy project has been using mypyc to compile mypy since 2019, giving it a 4x performance boost over regular Python.” [0] https://github.com/mypyc/mypyc https://github.com/mypyc/mypyc

derbOac 3y ago
... and also now there's Mojo too (https://www.modular.com/mojo https://www.modular.com/mojo). Not the same but seems relevant.
- maegul 3y ago
  Yea it’s gotten to the point where it’s a sub-domain of expertise in Python being on top of the various performance extensions/runtimes/hacks. Intuitively, numba and cython still feel like the most interesting/relevant. I recall talk a few years ago about the possibility of writing packages in “numba”. IE, targeting the subset of Python and numpy that numba supports. Occasionally I check in with numba’s releases and they seem to be slowly but surely supporting more of Python such that surely it will start to make sense to talk about the “numba” language. Has anyone got a tighter grip on whether this than I?
  - BiteCode_dev 3y ago
    I would say the most popular way to speed up a python program today is not through a compiler anymore, but using rust for hot loops and maturin (https://pypi.org/project/maturin/ https://pypi.org/project/maturin/) for seamlessly use it in Python and provide packages. I get the benefit of being able to "just use python" and still gain a speed up though. Plus there are many situations with rust is no possible (no time to learn such a complex language, rust is not approved by the security team).
    - tialaramex 3y ago
      The other edge is where your heatmap is just all red, and so "speed up hot loops" isn't going to get it done, and at that point somebody needs to bite the bullet and {hire people who know / learn} an AOT language with good performance. or it turns out the reason the heatmap is all red is that you were solving categorically the wrong problem, e.g. you decided to use machine vision to figure out what's in the packaging, while your competitors are just scanning the EAN-13 barcode.
- BiteCode_dev 3y ago
  Careful though, while Mojo is exciting, it's very tied to the modular platform for now, unlike the other projects.
- gkbrk 3y ago
  All the other options that were mentioned are open-source and available to public usage. They actually exist. Mojo on the other hand has an empty Github repo, no downloads, and no public access. You need to get approval from Modular Inc before you can even use a demo of Mojo. Clicking the "Get started with Mojo" or the "Request access" buttons take you to the same page with a form that asks your full name and email address, but giving them your personal info does not actually give you access to Mojo. It just gives you the following message until someone at Modular Inc vets your application. > We will contact you as soon as we are ready to onboard you into our early preview program. All the other options mentioned in this thread actually have downloads and repos that you can check out, and you don't need to fill an application form to "Request access". They exist, right now.
  - derbOac 3y ago
    Yeah I didn't mean to advocate for Mojo necessarily. I use Python for projects and like a lot about it, but often feel as if its implementations are kind of fragmented in many respects, and Mojo seems like it could potentially further that trend for me at least, at the current time. Other projects are certainly more open and established. For me personally what I want to see is a completely open, working, high-performance Python compiler that is fully compatible with official spec 100%, in the sense that you could take any code anywhere and it would work just as well on the official interpreter as on the compiler. Maybe this means creating an official Python 4.0 spec or something that's a superset of 3.0, or something, but right now I'm not sure I see anything quite like that. I could see Mojo being that eventually but you're right that at the moment it's far from that.

bloaf 3y ago

While we're at it: * Taichi "is embedded in Python and uses modern just-in-time (JIT) frameworks (for example LLVM, SPIR-V) to offload the Python source code to native GPU or CPU instructions, offering the performance at both development time and runtime." https://www.taichi-lang.org/ https://www.taichi-lang.org/

longqzh 3y ago
Very interested in the benchmark between Taichi and numba, both of them are using llvm as backend

prpl 3y ago

And Jax, and now Mojo, and while we’re at it, Graal.Python, and all sorts of other forgotten ones.

nologic01 3y ago

A wikipedia entry with a side by side comparison would seem appropriate at this stage

nurettin 3y ago

Cython lets you incrementally add types to your python code and transpile to C. Its best feature is the html output that highlights which part of your code still has an external dependency and is thus not directly translatable to C. Numba has an amazing jit decorator. It can even build on numpy functions. Once I had a simple sum loop and it was compiled into O(1). Pyston is very cool, it outputs C++ code that you can statically build into your project. Still needs libpython. Pypy speeds up plain python code. It can even install from pip. I tried it on neat-python package, solid 40% improvement. Nuitka is different. It is a packaging system which embeds the python interpreter and your code into a distributable binary.

helsinkiandrew 3y ago

Two questions: 1) Why so many Python compilers - should python be more open/modular so different JITs and other functionality can be plugged in/chosen at install or runtime? 2) Why has no one named a compiler "Monty"?

BiteCode_dev 3y ago
They don't have the same goals, nor characteristics. Numba and cython are rarely used to write the entire program, you usually apply them on your bottleneck, and compile just this part. Also JIT, like for numba or pypy, compile when the program runs, but projects like cython and nuikta compile ahead of time. The performance characteristics are very different, since the former need a warmup but the latter don't, however they need to be manually built. Solutions like cython or mypyc require type hinting for getting the best speed up. All of them except nuikta target better perfs, some in general (pyston, mypyc, cython), some targetting specifically numerical calculations (numba). nuitka doesn't care about any of that, it takes regular Python code and compiles it. While it does provide some speedup (up to 4x), it's not the goal at all. The goal of nuikta is to able you to ship your python program as a standalone executable to the end user. It's the most reliable way to do it, as, just like the tag line says, it's "extremly compatible". I've yet to find a Python program that nuikta couldn't compile, including one with PyQT + numpy. It's very good at it.

t-vi 3y ago

Not to forget CPython's own faster-cpython project which aims at JIT compiling. [0] Also JAX[1], PyTorch[2] come with JIT compilation specifically aimed at GPU kernels "fusing" multiple higher-level operation And NumPy/Scipy (also) uses Pythran[3], an AOT compiler not too unsimilar to Numba. [0] https://github.com/faster-cpython/cpython https://github.com/faster-cpython/cpython [1] https://pytorch.org/docs/stable/generated/torch.compile.html https://pytorch.org/docs/stable/generated/torch.compile.html [2] https://jax.readthedocs.io/ https://jax.readthedocs.io/ [3] https://pythran.readthedocs.io/ https://pythran.readthedocs.io/ I think some useful classification criteria would be - does it replace running code in Python (either own interpreter or compiler), vs does it speed up certain bits, - does it aim to faithfully implement Python or does it intentionally diverge in the semantics, - does it provide low-level semantics (where numba, pythran shine) or higher-level (e.g. what PyTorch, JAX do) - target architectures (CPU, GPU offloading, ...)