10 ms·

Nuitka: An extremely compatible Python compiler

223 points by BiteCode_dev 5y ago

asicsp 5y ago
Some of the past discussions: * https://news.ycombinator.com/item?id=8771925 https://news.ycombinator.com/item?id=8771925 (2014, 135 comments) * https://news.ycombinator.com/item?id=10994267 https://news.ycombinator.com/item?id=10994267 (2016, 52 comments) * https://news.ycombinator.com/item?id=15354613 https://news.ycombinator.com/item?id=15354613 (2017, 60 comments)
hultner 5y ago
Nuitka is a wonderful project which in my opinion doesn't get enough attention. I first found it back in 2015 when I worked at a company where we built a python based desktop application as a part of our industrial control system. Nuitka provided better performance then pyinstaller/cx_freeze while it was still simple to work with. Back then there were still a few incompatibilities but I've followed the project throughout the years and it's mature like a fine wine since.
aarchi 5y ago
Nuitka looks like a traditional ahead-of-time compiler using SSA form[0] and is written in Python. I’d be interested in seeing performance comparisons with PyPy, which uses the second Futamura protection and is written in a dialect of Python. [0]: https://nuitka.net/doc/developer-manual.html#ssa-form-for-nuitka https://nuitka.net/doc/developer-manual.html#ssa-form-for-nu...
- BiteCode_dev 5y ago
  It's slower than pypy, but starts faster and results in a standalone executable. Also, it's compatible with most c extensions.
- igouy 5y ago
  https://pybenchmarks.org/u64q/benchmark.php?test=all&lang=nuitka&lang2=pypy&data=u64q https://pybenchmarks.org/u64q/benchmark.php?test=all&lang=nu... "Python Interpreters Benchmarks"
  - tudelo 5y ago
    Thanks for the link. Don't know if you are affiliated but there is a lot of link rot going on there with external links.
    - igouy 5y ago
      Sorry, not mine. https://github.com/dundee/pybenchmarks/issues https://github.com/dundee/pybenchmarks/issues
m4r35n357 5y ago
Unlike most Python "accelerators" this seems to work OOTB with no faff!
nextlevelwizard 5y ago
How is this more compatible than just python? Since it requires libpython so you anyway need to have python installed on the target machine? Also is this actually faster than just python? I tried it out with some bad expensive looping using lists with python3.9 and it was 100ms slower (1.3sec vs 1.4sec). Just in general why should I be using Nuitka?
- nightcracker 5y ago
  I think the idea behind Nuitka is to compile Python into an executable, rather than interpret a script using a fully generic interpreter, but without (or with minimal) limitations, falling back to libpython's implementation if necessary. > In the future Nuitka will be able to use type inferencing based on whole program analysis. It will apply that information in order to perform as many calculations as possible in C, using C native types, without accessing libpython. I believe the difference with PyPy is that PyPy tries to do this using just-in-time compilation and nuitka uses ahead-of-time compilation.
- BiteCode_dev 5y ago
  nuitka takes your python code and all its deps, turns it to c code, and compiles that into a standalone executable that is both faster and doesn't need a separate python vm to be installed.
  - nextlevelwizard 5y ago
    Expect it does require libpython and at least on Ubuntu 20.04 it links against the dev version of the library. So not only can you not run the resulting binary on any random Ubuntu 20.04 you also need to install same version of the python and the "-dev" version. I did try this. With the "stock" python3 (3.8.something) I couldn't get nuitka to compile my code (or the example from the docs). With python3.9 I could start the compilation, but I didn't have libpython-dev shared object and finally with python3.9-dev I was able to make the binary. However after transfering it to another Ubuntu 20.04 machine I couldn't run it since it was linked against libpython3.9-dev. Just as side note. The same python code (i.e. the .py file) would run on both machines out of the box.
philpem 5y ago
Wow, this got my attention. I use Python for web application development. A 2-3x speedup would be very interesting for higher-load deployments.
- qeternity 5y ago
  You're extremely unlikely to get much, if any, speed up on a large, general codebase. The 2-3x improvements are on specific micro benchmarks.
- BiteCode_dev 5y ago
  The best feature of nuitka, and its main goal, is not speed gain. That's the cherry on top. The main goal is getting a standalone executable that works with c extensions.
  - synergy20 5y ago
    that's correct, nuitka is to package python into one single executable, it's an 'installer' per se.
    - BiteCode_dev 5y ago
      Not an installer no. The installer part is not provided, the result is an executable that works out of the box. The installer would then take that and put it somewhere, but you don't need to. It can even work from a USB key.
  - antman 5y ago
    Does it also package the extensions?
- dec0dedab0de 5y ago
  if you're just looking to speed things up PyPy, and Cython are both good options
  - vips7L 5y ago
lennoff 5y ago
Tangentially related question: could Nuitka target WebAssembly, given that according to their overview page, they translate Python into C? Usually when it comes to Python -> WebAssembly, the biggest problem is the lack of GC in the WebAssembly spec (as far as I understand), and I'm wondering if this would be an issue for Nuitka as well...
- codeflo 5y ago
  Of course you can build a GC in WebAssembly. You might have to avoid the native stack and lose some performance that way, but that shouldn’t be too bad I think. One problem is that this GC doesn’t interact with the browser’s GC in any way. So you have painful memory management interactions when (for example) a DOM event handler references a WebAssembly object which in turn holds a reference to a browser object, possibly with cycles (so simple reference counting isn’t enough). You can fall back to manual memory management here, but that’s painful to use. To make this work seamlessly, you need a way to trace references across both heaps in one swoop. Last I checked, there was standardization work underway to enable that.
- BiteCode_dev 5y ago
  Yes but pyodide is probably better suited for that and can load uncompiled python code. See : https://github.com/pyodide/pyodide https://github.com/pyodide/pyodide And an example featuring a pure client side jupyter instance : https://news.ycombinator.com/item?id=28377550 https://news.ycombinator.com/item?id=28377550
eatonphil 5y ago
This project is super impressive. It makes me wonder why there's not a similarly mature JavaScript AOT compiler implementation for use in game development or other places where you want performance better than a VM and you're not allowed to JIT. A small recommendation for these release pages would be to link to the tag in whatever source host you're using! I just want to see the code.
- maattdd 5y ago
  There is, for example https://bellard.org/quickjs/ https://bellard.org/quickjs/
  - eatonphil 5y ago
    That's an interpreter written in C and without a JIT. Not the same as an AOT compiler. https://github.com/bellard/quickjs/blob/master/quickjs-opcode.h https://github.com/bellard/quickjs/blob/master/quickjs-opcod... The point isn't just not having a JIT. You can run v8 without a JIT. The point is good performance without a JIT.
    - BiteCode_dev 5y ago
      "Can compile Javascript sources to executables with no external dependency"
      - eatonphil 5y ago
        > @item -c > Only output bytecode in a C file. The default is to output an executable file. > @item -e > Output @code{main()} and bytecode in a C file. The default is to output an executable file. https://github.com/bellard/quickjs/blob/master/doc/quickjs.texi#L159 https://github.com/bellard/quickjs/blob/master/doc/quickjs.t...
      - formerly_proven 5y ago
        Which probably works by stapling bytecode to a precompiled interpreter. A tried and true strategy, but not necessarily what people want when asking for AOT compilation.
infocollector 5y ago
If you used pip install nutika -U - on both mac and linux, it doesn't find an installation candidate for me. pip install -U "https://github.com/Nuitka/Nuitka/archive/develop.zip https://github.com/Nuitka/Nuitka/archive/develop.zip" works on both currently. But this does not get "nutika" on the path. Perhaps someone here is more familiar with the installation system and can comment?
- scbrg 5y ago
  nutika ≠ nuitka
- BiteCode_dev 5y ago
  It's nuitka, not nutika. And use "python -m nuitka", not "nuitka" alone. "-m" is a very important option in python that solves a lot of path problems.
aitchnyu 5y ago
Does it print the same stacktrace as CPython?
- BiteCode_dev 5y ago
  Yes, but since nuitka targets distribution, you probably wants to disable that for the end user.
BiteCode_dev 5y ago
See also: shiv, which doesn't compile, but bundles you python scripts and deps: https://news.ycombinator.com/item?id=28377545 https://news.ycombinator.com/item?id=28377545 Very nice for deployment and scripting.
- stavros 5y ago
  This looks nice for distribution, thanks!
eatonphil 5y ago
It would be neat if the big companies like Dropbox, Instagram, Google, Oracle, Shopify, Stripe, whatnot building better Python/JavaScript/Ruby implementations would start building program analysis libraries for Python/JavaScript/Ruby so that more implementations could get this for free. For example, the homepage of Nuitka says they only just added support for constant folding and propagation. That's such low hanging fruit it's crazy they have to do that for themselves. You can imagine generic libraries for Python/JavaScript/Ruby that will turn the AST alone into its most optimal form and then let some other backend worry about code generation or VM implementation.
- shadowfox 5y ago
  An issue, though not an unsolvable one, is that you probably need a standardized representation of the AST to write these optimization passes against; it is not uncommon for compiler writers to disagree on what representation is the best for what purpose. Then there is the question of optimizations that are (typically) easier/possible only at the code-generation phase. All of this is solvable of course, but there needs to be some will to do so (or in the case of companies, enough commercial benefit).
  - zeckalpha 5y ago
    LLVM IR?
metalliqaz 5y ago
How well does it handle tkinter?
adsharma 5y ago
Extreme compatibility has a cost. If you're willing to give up some compatibility, you could get faster code that can be checked statically by transpiling to one of the many great choices from a set of statically typed languages. This is the approach taken by py14, pyrs and their successor py2many, on which I've been working for some months.
- nextaccountic 5y ago
  But it was the lack of 100% compatibility that hampered pypy's adoption. Anyway what are your incompatibilities?
  - int_19h 5y ago
    It seems to be fully statically typed, for starters: https://github.com/adsharma/py2many/blob/main/doc/langspec.md https://github.com/adsharma/py2many/blob/main/doc/langspec.m... But, given LuaJIT and V8, it's obviously possible to generate high-performance code without such restrictions. As for PyPy adoption, it's hindered more by the lack of API compatibility with CPython native extensions.
    - adsharma 5y ago
      Yes - Julia also has a pretty good JIT too, but doesn't have a great story on AOT compiled binaries. If you're finding it easy to write a quick command line tool in python, but end up rewriting it in another language for size/performance/self contained deployment, this may be a good fit.
  - adsharma 5y ago
    Doesn't use python's C API or runtime. So no metaclasses, monkey patching, eval(). All types must be known at compile time or it should be possible to infer them. There are several challenges to solve - translating stdlib of one language to another (see plugins.py) and bridge semantic gaps (rust doesn't like mutable global state, python has them).
    - 5y ago
knxl 5y ago
only one thing to say. "compatibility comes at a cost."
keithasaurus 5y ago
If you're already using Python's type hints, I'd suggest checking out mypyc instead. If your code is already type-checked by mypy, you don't need to do much more than run `mypyc` to get a performance boost.
antman 5y ago
How does this compare with pyston which was praised in previous HN discussions?
ficklepickle 5y ago
> Retry downloads without SSL if that fails, as some Python do not have working SSL. Fixed in 0.6.14.5 already. Is this a good idea? If I understand correctly, if an HTTPS request fails, it falls back to HTTP. So an attacker could just block the HTTPS request, then MITM the HTTP request.

223 points by BiteCode_dev 5y ago

Some of the past discussions: * https://news.ycombinator.com/item?id=8771925 https://news.ycombinator.com/item?id=8771925 (2014, 135 comments) * https://news.ycombinator.com/item?id=10994267 https://news.ycombinator.com/item?id=10994267 (2016, 52 comments) * https://news.ycombinator.com/item?id=15354613 https://news.ycombinator.com/item?id=15354613 (2017, 60 comments)

hultner 5y ago

Nuitka is a wonderful project which in my opinion doesn't get enough attention. I first found it back in 2015 when I worked at a company where we built a python based desktop application as a part of our industrial control system. Nuitka provided better performance then pyinstaller/cx_freeze while it was still simple to work with. Back then there were still a few incompatibilities but I've followed the project throughout the years and it's mature like a fine wine since.

aarchi 5y ago

Nuitka looks like a traditional ahead-of-time compiler using SSA form[0] and is written in Python. I’d be interested in seeing performance comparisons with PyPy, which uses the second Futamura protection and is written in a dialect of Python. [0]: https://nuitka.net/doc/developer-manual.html#ssa-form-for-nuitka https://nuitka.net/doc/developer-manual.html#ssa-form-for-nu...

BiteCode_dev 5y ago
It's slower than pypy, but starts faster and results in a standalone executable. Also, it's compatible with most c extensions.
igouy 5y ago
https://pybenchmarks.org/u64q/benchmark.php?test=all&lang=nuitka&lang2=pypy&data=u64q https://pybenchmarks.org/u64q/benchmark.php?test=all&lang=nu... "Python Interpreters Benchmarks"
- tudelo 5y ago
  Thanks for the link. Don't know if you are affiliated but there is a lot of link rot going on there with external links.
  - igouy 5y ago
    Sorry, not mine. https://github.com/dundee/pybenchmarks/issues https://github.com/dundee/pybenchmarks/issues

m4r35n357 5y ago

Unlike most Python "accelerators" this seems to work OOTB with no faff!

nextlevelwizard 5y ago

How is this more compatible than just python? Since it requires libpython so you anyway need to have python installed on the target machine? Also is this actually faster than just python? I tried it out with some bad expensive looping using lists with python3.9 and it was 100ms slower (1.3sec vs 1.4sec). Just in general why should I be using Nuitka?

nightcracker 5y ago
I think the idea behind Nuitka is to compile Python into an executable, rather than interpret a script using a fully generic interpreter, but without (or with minimal) limitations, falling back to libpython's implementation if necessary. > In the future Nuitka will be able to use type inferencing based on whole program analysis. It will apply that information in order to perform as many calculations as possible in C, using C native types, without accessing libpython. I believe the difference with PyPy is that PyPy tries to do this using just-in-time compilation and nuitka uses ahead-of-time compilation.
BiteCode_dev 5y ago
nuitka takes your python code and all its deps, turns it to c code, and compiles that into a standalone executable that is both faster and doesn't need a separate python vm to be installed.
- nextlevelwizard 5y ago
  Expect it does require libpython and at least on Ubuntu 20.04 it links against the dev version of the library. So not only can you not run the resulting binary on any random Ubuntu 20.04 you also need to install same version of the python and the "-dev" version. I did try this. With the "stock" python3 (3.8.something) I couldn't get nuitka to compile my code (or the example from the docs). With python3.9 I could start the compilation, but I didn't have libpython-dev shared object and finally with python3.9-dev I was able to make the binary. However after transfering it to another Ubuntu 20.04 machine I couldn't run it since it was linked against libpython3.9-dev. Just as side note. The same python code (i.e. the .py file) would run on both machines out of the box.

philpem 5y ago

Wow, this got my attention. I use Python for web application development. A 2-3x speedup would be very interesting for higher-load deployments.

qeternity 5y ago
You're extremely unlikely to get much, if any, speed up on a large, general codebase. The 2-3x improvements are on specific micro benchmarks.
BiteCode_dev 5y ago
The best feature of nuitka, and its main goal, is not speed gain. That's the cherry on top. The main goal is getting a standalone executable that works with c extensions.
- synergy20 5y ago
  that's correct, nuitka is to package python into one single executable, it's an 'installer' per se.
  - BiteCode_dev 5y ago
    Not an installer no. The installer part is not provided, the result is an executable that works out of the box. The installer would then take that and put it somewhere, but you don't need to. It can even work from a USB key.
- antman 5y ago
  Does it also package the extensions?
dec0dedab0de 5y ago
if you're just looking to speed things up PyPy, and Cython are both good options
- vips7L 5y ago

lennoff 5y ago

Tangentially related question: could Nuitka target WebAssembly, given that according to their overview page, they translate Python into C? Usually when it comes to Python -> WebAssembly, the biggest problem is the lack of GC in the WebAssembly spec (as far as I understand), and I'm wondering if this would be an issue for Nuitka as well...

codeflo 5y ago
Of course you can build a GC in WebAssembly. You might have to avoid the native stack and lose some performance that way, but that shouldn’t be too bad I think. One problem is that this GC doesn’t interact with the browser’s GC in any way. So you have painful memory management interactions when (for example) a DOM event handler references a WebAssembly object which in turn holds a reference to a browser object, possibly with cycles (so simple reference counting isn’t enough). You can fall back to manual memory management here, but that’s painful to use. To make this work seamlessly, you need a way to trace references across both heaps in one swoop. Last I checked, there was standardization work underway to enable that.
BiteCode_dev 5y ago
Yes but pyodide is probably better suited for that and can load uncompiled python code. See : https://github.com/pyodide/pyodide https://github.com/pyodide/pyodide And an example featuring a pure client side jupyter instance : https://news.ycombinator.com/item?id=28377550 https://news.ycombinator.com/item?id=28377550

eatonphil 5y ago

This project is super impressive. It makes me wonder why there's not a similarly mature JavaScript AOT compiler implementation for use in game development or other places where you want performance better than a VM and you're not allowed to JIT. A small recommendation for these release pages would be to link to the tag in whatever source host you're using! I just want to see the code.

maattdd 5y ago
There is, for example https://bellard.org/quickjs/ https://bellard.org/quickjs/
- eatonphil 5y ago
  That's an interpreter written in C and without a JIT. Not the same as an AOT compiler. https://github.com/bellard/quickjs/blob/master/quickjs-opcode.h https://github.com/bellard/quickjs/blob/master/quickjs-opcod... The point isn't just not having a JIT. You can run v8 without a JIT. The point is good performance without a JIT.
  - BiteCode_dev 5y ago
    "Can compile Javascript sources to executables with no external dependency"
    - eatonphil 5y ago
      > @item -c > Only output bytecode in a C file. The default is to output an executable file. > @item -e > Output @code{main()} and bytecode in a C file. The default is to output an executable file. https://github.com/bellard/quickjs/blob/master/doc/quickjs.texi#L159 https://github.com/bellard/quickjs/blob/master/doc/quickjs.t...
    - formerly_proven 5y ago
      Which probably works by stapling bytecode to a precompiled interpreter. A tried and true strategy, but not necessarily what people want when asking for AOT compilation.

infocollector 5y ago

If you used pip install nutika -U - on both mac and linux, it doesn't find an installation candidate for me. pip install -U "https://github.com/Nuitka/Nuitka/archive/develop.zip https://github.com/Nuitka/Nuitka/archive/develop.zip" works on both currently. But this does not get "nutika" on the path. Perhaps someone here is more familiar with the installation system and can comment?

scbrg 5y ago
nutika ≠ nuitka
BiteCode_dev 5y ago
It's nuitka, not nutika. And use "python -m nuitka", not "nuitka" alone. "-m" is a very important option in python that solves a lot of path problems.

aitchnyu 5y ago

Does it print the same stacktrace as CPython?

BiteCode_dev 5y ago
Yes, but since nuitka targets distribution, you probably wants to disable that for the end user.

BiteCode_dev 5y ago

See also: shiv, which doesn't compile, but bundles you python scripts and deps: https://news.ycombinator.com/item?id=28377545 https://news.ycombinator.com/item?id=28377545 Very nice for deployment and scripting.

stavros 5y ago
This looks nice for distribution, thanks!

eatonphil 5y ago

It would be neat if the big companies like Dropbox, Instagram, Google, Oracle, Shopify, Stripe, whatnot building better Python/JavaScript/Ruby implementations would start building program analysis libraries for Python/JavaScript/Ruby so that more implementations could get this for free. For example, the homepage of Nuitka says they only just added support for constant folding and propagation. That's such low hanging fruit it's crazy they have to do that for themselves. You can imagine generic libraries for Python/JavaScript/Ruby that will turn the AST alone into its most optimal form and then let some other backend worry about code generation or VM implementation.

shadowfox 5y ago
An issue, though not an unsolvable one, is that you probably need a standardized representation of the AST to write these optimization passes against; it is not uncommon for compiler writers to disagree on what representation is the best for what purpose. Then there is the question of optimizations that are (typically) easier/possible only at the code-generation phase. All of this is solvable of course, but there needs to be some will to do so (or in the case of companies, enough commercial benefit).
- zeckalpha 5y ago
  LLVM IR?

metalliqaz 5y ago

How well does it handle tkinter?

adsharma 5y ago

Extreme compatibility has a cost. If you're willing to give up some compatibility, you could get faster code that can be checked statically by transpiling to one of the many great choices from a set of statically typed languages. This is the approach taken by py14, pyrs and their successor py2many, on which I've been working for some months.

nextaccountic 5y ago
But it was the lack of 100% compatibility that hampered pypy's adoption. Anyway what are your incompatibilities?
- int_19h 5y ago
  It seems to be fully statically typed, for starters: https://github.com/adsharma/py2many/blob/main/doc/langspec.md https://github.com/adsharma/py2many/blob/main/doc/langspec.m... But, given LuaJIT and V8, it's obviously possible to generate high-performance code without such restrictions. As for PyPy adoption, it's hindered more by the lack of API compatibility with CPython native extensions.
  - adsharma 5y ago
    Yes - Julia also has a pretty good JIT too, but doesn't have a great story on AOT compiled binaries. If you're finding it easy to write a quick command line tool in python, but end up rewriting it in another language for size/performance/self contained deployment, this may be a good fit.
- adsharma 5y ago
  Doesn't use python's C API or runtime. So no metaclasses, monkey patching, eval(). All types must be known at compile time or it should be possible to infer them. There are several challenges to solve - translating stdlib of one language to another (see plugins.py) and bridge semantic gaps (rust doesn't like mutable global state, python has them).
  - 5y ago

knxl 5y ago

only one thing to say. "compatibility comes at a cost."

keithasaurus 5y ago

If you're already using Python's type hints, I'd suggest checking out mypyc instead. If your code is already type-checked by mypy, you don't need to do much more than run `mypyc` to get a performance boost.

antman 5y ago

How does this compare with pyston which was praised in previous HN discussions?

ficklepickle 5y ago

> Retry downloads without SSL if that fails, as some Python do not have working SSL. Fixed in 0.6.14.5 already. Is this a good idea? If I understand correctly, if an HTTPS request fails, it falls back to HTTP. So an attacker could just block the HTTPS request, then MITM the HTTP request.