17 ms·

Nuitka: a Python compiler written in Python

502 points by stunt 5y ago

KaiserPro 5y ago
This looks rather fun, has anyone here used it in anger?
- aogl 5y ago
  I wrote about this over a year ago over here https://ao.ms/how-to-package-a-python-app-using-nuitka/ https://ao.ms/how-to-package-a-python-app-using-nuitka/ I've also used it in a production estate, and it works wonderfully well. Would recommend!
  - KaiserPro 5y ago
    smashing, thanks for this.
- rvieira 5y ago
  I've used it for some simple, personal usage CLI tools and simple REST services. Works out-of-the-box and really well. From my experience some libraries are hard to compile (e.g. BeautifulSoup) but not impossible they need some tweaking. Sometimes I struggle with embedding data for some reason (especially since I'm spoiled with Go 1.16 `// embed`).
Yajirobe 5y ago
Nuitka is the best python compiler I've used. I have tried at least three others and none worked as good as nuitka. And I'm talking about complicated dependencies such as pytorch, selenium, tesseract - that type of stuff.
- FartyMcFarter 5y ago
  How do the resulting executables perform, compared to using CPython? Did you happen to run any benchmarks?
  - avinashv 5y ago
    Not the OP, but it claims to be ~2x faster than CPython. I haven’t done extensive benchmarking, but for my small projects that seems about right.
    - dagw 5y ago
      Based on my own tests, once you have a 'normal' program that does a bit of IO, calls out some C based libraries etc. etc. I've rarely seen more than 10-20% performance increase across the whole program run. For individual functions you can see 2-4 times speed up over pure python. Most importantly, I've never seen it result in slower than cpython performance.
    - jhgb 5y ago
      That sounds significantly slower than PyPy on average, which would hardly make it "the best python compiler". YMMV, of course.
      - throwaway894345 5y ago
        It's all about compatibility to be honest. PyPy has done yeoman's work and it's still lacking in compatibility for important packages (e.g., psycopg2). Performance means little if we can't use major parts of the ecosystem.
        jhgb 5y ago
        Is it preferable on PyPy to use a libpq wrapper over a native solution? As far as I can tell, even though CPython C API is in some way supported on PyPy it's really only advisable to use packages using it as a last resort for performance reasons. Aside from psycopg2cffi, which you should be able to use with PyPy, I'm pretty sure I saw other PEP 249 implementations for PostgreSQL the last time I checked (which admittedly was a few years ago).
        throwaway894345 5y ago
- mimimi31 5y ago
  >complicated dependencies such as pytorch, selenium, tesseract When I last tried Nuitka I couldn't get it to work with pycryptodome. Sounds like I should maybe give it another go.
  - gota 5y ago
    I had a similar error. If I recall correctyl, I had to copy the contents of lib/Crypto into the nuitka dist folder, and then it worked.
- neolog 5y ago
  What the heck use case is this?
truncate 5y ago
This looks pretty cool. A better summary at the top would be nice to new users. I had to scroll down a bit, and read few lines until I realized it translates Python to C, and then uses CC to do final compilation (it is also quite ambiguous if executable is native or interpreted, until you read that part). I'd be looking more into this, but anyone else knows any other dynamically typed language which compiles to native code, not JITed or bytecode (I remember there were few Lisps out there).
- astrange 5y ago
  There are some for Perl in various degrees of emitting bytecode to actually optimizing (perlcc, RPerl). I think others prefer emitting to Java or making their own JITs.
- scarygliders 5y ago
  My one claim to fame is that a few years ago, I suggested the TL;DR wording on the Nuitka Overview page: https://nuitka.net/pages/overview.html https://nuitka.net/pages/overview.html If I could find the message I sent to the Nuitka mailing list (or did I suggest it via email? Can't remember) I'd publish that here. ;)
  - scarygliders 5y ago
    ^ Seriously? Downvotes? What did I say which was wrong? Honestly thought it was a mildly interesting anecdote! Sheesh! You people! :)
- cestith 5y ago
  Steel Bank Common Lisp compiles Lisp to executables. Julia supports both a REPL and AOT compilation. There are some Forth compilers for various systems. https://www.thefreecountry.com/compilers/forth.shtml https://www.thefreecountry.com/compilers/forth.shtml
  - eesmith 5y ago
    Humorously, https://cliki.net/Python https://cliki.net/Python - "Python is the name of a free high-performance compiler originally developed at Carnegie-Mellon University for CMU Common Lisp, it is now used by SBCL ..."
foepys 5y ago
If you are looking for something lighter: CPython can execute ZIPs with a Python program inside. You can also prepend the python shebang before the ZIP and mark it as executable in the filesystem. youtube-dl gets distributed like that.
- randlet 5y ago
  The problem with this method is that the user still needs to have Python on their system. Pyinstaller (Py2exe, cx_Freeze) bundle the python interpreter along with your app.
  - londt8 5y ago
    I think its typical that only one python runtime is installed in the system. What is the use case for bundling the runtime with the app?
    - scarygliders 5y ago
      I package Captain's Log [0] with PyInstaller. I need to, because otherwise I wouldn't be able to package it for Windows users. The app is also closed source. I cannot easily package the application for Linux users, because of so many different flavours of Linux, so many different combinations of Python and Qt versions. Which is why I've given up on a pure Linux version and I'm in the middle of adapting it to work under WINE alongside Elite: Dangerous also running under WINE, instead. [0] https://captainslog.scarygliders.net/captains-log-2/ https://captainslog.scarygliders.net/captains-log-2/
    - nsomaru 5y ago
      Distribute my cli app + gooey as a stand-alone binary for windows users
    - curryst 5y ago
      Many systems still have both Python 2 and Python 3 installed. Also, in the standard case, the version of Python is controlled by the user, not by your app. Bundling it with the app lets you make sure that it's running Python 3.8, for example, so that you can use Python 3.8 features without having to add a bunch of compatibility shims in case the user is running an older version.
rlampert 5y ago
It is instructive to read GvR's take on Nuitka: https://news.ycombinator.com/item?id=8772124 https://news.ycombinator.com/item?id=8772124 GvR and the old boys currently present themselves as CoC proponents, and strategically use the CoC against anyone who criticizes them. Compare with his current Microsoft sponsored activities. If anyone would use that language against "his" (i.e. Mark Shannon's) project, they'd be out and suffer public defamation in no time.
- wly_cdgr 5y ago
  Wow, didn't realize he could be such a baby Edit: wow GvR's pals and fanboys are salty about someone pointing out what a loser he can be
- gedazz 5y ago
  What does CoC stand for?
  - isp 5y ago
    CoC = "Code of Conduct" I haven't followed Python dev, so can't comment about how the CoC is being used in practice.
    - WesolyKubeczek 5y ago
      > I haven't followed Python dev, so can't comment about how the CoC is being used in practice. In tough situations, CoC powers are inversely proportional to the height of your position on the totem pole.
  - Joker_vD 5y ago
    Still can't believe that this abbreviation is actually in use without any objections while there are talks about needing to rename the Coq language.
    - shpongled 5y ago
      I imagine that most people saying "CoC" out loud are not saying "cock", instead either "Code of conduct", or "cee oh cee".
myWindoonn 5y ago
I wish Nuitka were a drop-in solution, like PyPy, but it seems like I've never had a project that is 100% Nuitka-compatible.
- ok123456 5y ago
  Any scientific computing application that uses numpy, that is the vast majority of them, will not work with nuitka.
  - gota 5y ago
    Hey, I did just that last week (actually I import Pandas, which requires numpy). There's a "plugin" option for numpy, in Nuitka, that makes it work. There are similar plugins for a handful of packages.
    - gota 5y ago
      For reference, the page with the plugins' documentation is not a very top search result, so I'll include it here: https://github.com/Nuitka/Nuitka/blob/develop/Standard-Plugins-Documentation.rst https://github.com/Nuitka/Nuitka/blob/develop/Standard-Plugi... The list of plugins is: dill-compat Required by the dill module eventlet Required by the eventlet package gevent Required by the gevent package multiprocessing Required by Python's multiprocessing module numpy Required for numpy, scipy, pandas, matplotlib, etc. pmw-freezer Required by the Pmw package pylint-warnings Support PyLint / PyDev linting source markers qt-plugins Required by the PyQt and PySide packages tensorflow Required by the tensorflow package tk-inter Required by Python's Tk modules torch Required by the torch / torchvision packages
gardnr 5y ago
Does this mean python is now self-hosted?
- neolog 5y ago
  Python is self-hosted under PyPy.
shivekkhurana 5y ago
I used Nuitka many years ago, good to see it on the front page.
BlankReg 5y ago
I've used Nuitka a couple times for some commercial Raspberry Pi projects and it worked perfectly each time.
Operyl 5y ago
Another cool project like this is https://github.com/Pypperoni/ https://github.com/Pypperoni/ but it takes a bit more of your own set up to get going. Disclaimer: just a happy user.
pansa2 5y ago
Is Nuitka able to create self-contained binaries that can be distributed as a single file? Or do they still have external dependencies such as the Python runtime? I’ve chosen Go rather than Python for a few small projects recently. Not for performance, parallelism or a particular fondness for the language, but just because Go can build truly standalone executables.
- reacharavindh 5y ago
  The last time I tried this was quite a while ago and your question was exactly why I tried. But, it failed with some unintuitive errors in dependencies of my Python script, and I gave up after 10 mins of Googling. I’m still wary of that experience and will avoid Python where I have such deployment needs unless the language natively comes up with such a build solution. FWIW, I think borg(backup solution written in Python and C) uses pyinstaller to get a single binary executable. It may be of interest to you.
- thinker5555 5y ago
  According to the readme: > The created binaries can be made executable independent of the Python installation, with --standalone and --onefile options.
- vascocosta 5y ago
  I use Pyinstaller to generate single file executables of my Python projects, which use Qt. It works great.
  - XzAeRosho 5y ago
    The only problem I have found with Pyinstaller is that their "standalone" builds still depends on certain libraries that will change with the version of Python in the host OS, such as the glibc library, or others. The proposed solution by Pyinstaller is to build your program in the "oldest" version you support, which is silly compared to build systems like Go's, which are true standalone. Having said that, I'm curious about this solution, since it seems to claim a true standalone build...
jftuga 5y ago
How does this compare with PyInstaller?
- woodrowbarlow 5y ago
  pyinstaller doesn't compile your code. instead, it bundles a copy of the python interpreter along with your code, and sets everything up with an executable header so you can double-click the file to run your code within the embedded interpreter.
liveoneggs 5y ago
anyone tried GraalVM Python for this?
- anentropic 5y ago
  I looked into the this question for Graal Nodejs and I think it has the same problem in Python... Basically Graal Python and Nodejs each provide a custom interpreter for the target lang, the main goal of which is to provide interoperability with the Graal polyglot ecosystem. So you can run your python code under the GraalPython interpreter and it will run JITed fast and can import libs from other Graal-supported langs. But as far as outputting executable binaries, Graal only provides that for JVM projects and LLVM languages like C/C++/Rust. So it's not impossible, but you have to build your own Java wrapper project that loads the Graal Python interpreter class in code and then runs your python lib inside that. I expect eventually that boilerplate step can be automated as part of the Graal build tools. If I've got this wrong I welcome any corrections!
  - anentropic 5y ago
    I was half-remembering what I found when researching Graal Nodejs clearer, less-garbled version as an answer here: https://stackoverflow.com/a/67331258/202168 https://stackoverflow.com/a/67331258/202168
Cadwhisker 5y ago
I've not tried Nuitka yet, but I've done similar things with 2 other tools to package up a Qt-based GUI tool that has a Windows installer, Mac ".dmg" or a Linux package: * FBS (FMan's Build System) - worked well for a while, but we couldn't upgrade Qt past a certain version and he version we used was only happy with Python 3.6. * Beeware - our current solution. Very, very happy with this; works smoothly, lets us use the latest Python and Qt versions and lets us make professional-looking installers that we can distribute.
- mherrmann 5y ago
  fbs author here; The Pro version of fbs [1] now supports Python > 3.6 and recent Qt versions. 1: https://build-system.fman.io https://build-system.fman.io
  - diogenesjunior 5y ago
    What I love about HN. You can discuss almost anything and the project's creator pops up in the thread.
  - Cadwhisker 5y ago
    Ah, great news. Thanks for making FBS; it is incredibly liberating to find a tool that lets us write Python GUI apps and have them appear as if they were made by XCode or Visual Studio.
    - mherrmann 5y ago
      Happy if you find it useful!
  - sireat 5y ago
    Nice option for building Python+Qt applications! Buying the Pro version still leaves one to deal with Qt licensing does it not?
    - mherrmann 5y ago
brenainn 5y ago
I remember this being promising but struggling with a dependency that was using entry points to dynamically load plugins, and I couldn't get Nuitka to register that. Seeing this is a good reminder to have another shot at it.
jll29 5y ago
hello.py 1-liner on Ubuntu Linux: $ python -m pip install nuitka $ python3.8 -m nuitka hello.py install Nuitka with no problems and compiled "hello world", also with no problems. It created a 4,060,104 byte hello.bin with no symbol table, so no need to run strip to make it smaller, which again worked like a breeze. (The developers say on GutHub they will work on smarter ways to keep the dependency list small, which can at present escalate for modules like pandas that themselves have >1000 dependencies.)
zoom6628 5y ago
Have used on home projects couple of years ago to compile code on macOS and run on Windows and on Debian(on a pi). Worked well. Looking forward to retrying nuitka again once my current project gets to point I've finished the firmware and get back to working on middleware. Def worth a look for anybody that wants to distribute python developed system without installing python or the inevitable many dependencies of projects. It just makes distribution a whole lot more predictable. And to the reader who asked about GraalPython its on my todo list - have it installed but life, children, and work keep getting in the way.
jonnycomputer 5y ago
I suppose if my target is on x version of y OS then I'll have to compile on that machine, yeah? Sorry, I don't have much experience creating these sort of packages for distribution, but have recently found the need.
- killingtime74 5y ago
  It depends on the language. Could require it or could not. Not sure about here.
samsquire 5y ago
I love Python so much. It's my favourite language. It's so easy. I don't even use the dynamic nature of Python - monkey patching etc. I wish more people wrote algorithms in Python for understandability then compiled them into a compiled language for performance. I find that Python resembles pseudocode because it is syntactically sparse, unlike C++. In Python I don't need to worry about ownership or memory. It probably needs Pointers to be a low level system language. I just wish Python had parallelism.
- divbzero 5y ago
  By parallelism do you mean something different from multiprocessing? [1] [1]: https://docs.python.org/3/library/multiprocessing.html https://docs.python.org/3/library/multiprocessing.html
  - triceratops 5y ago
    Pretty sure they're referring to the GIL.[1] 1. https://realpython.com/python-gil/ https://realpython.com/python-gil/
  - samsquire 5y ago
    I want Thread to run in parallel. I've tried multiprocessing but I found it buggy for my use case. I tried to create a worker queue and submit items to worker threads with JoinableQueue but eventually the system deadlocks and no progress is made. I'm not sure what the problem is. It never finishes. It's a sentence correlator using multiprocessing - it generates correlations of words in your sentences. https://github.com/samsquire/notebook/blob/master/sentence-correlator.py https://github.com/samsquire/notebook/blob/master/sentence-c... https://github.com/samsquire/notebook/blob/master/workers.py https://github.com/samsquire/notebook/blob/master/workers.py Maybe somebody can catch what I'm doing wrong. edit: it may be because i've forgotten to run task_done on joinable queue. edit2: yes it was that! it works!
    - aw1621107 5y ago
      Out of curiosity, is using multiprocessing.Pool’s map/etc.() methods not an option? It’s a higher-level API which would remove the need to manually manage workers/a queue.
jokoon 5y ago
On their website: > Future > In the future Nuitka will be able to use type inferencing based on whole program analysis. It will apply that information in order to perform as many calculations as possible in C, using C native types, without accessing libpython. They already claim a 300% speed boost, but with those type inferencing things, it will probably go a lot of further... Can't wait for this project to mature. I guess compiling to WebAssembly should be trivial then?
- codethief 5y ago
  Speaking of speedup and program analysis, there's also mypyc[0] which relies on type annotations and also claims a performance boost: > The mypy project has been using mypyc to compile mypy since 2019, giving it a 4x performance boost over regular Python. Does anyone know how mypyc and Nuitka compare in practice? [0]: https://github.com/mypyc/mypyc https://github.com/mypyc/mypyc
cduzz 5y ago
What's the advantage of this over docker? Presumably this just bundles a python runtime and a copy of the dependencies and hides it all inside some sort of huge file?
- ebrewste 5y ago
  I have a project that pyinstaller can make into a 12 MB single file. Docker turns that same project into a ~400 MB container. This is not considering the Docker install footprint.
sscarduzio 5y ago
Amazing idea, is there anything like this for Javascript?
dang 5y ago
Past related threads: Nuitka 0.6.0 released - https://news.ycombinator.com/item?id=18092837 https://news.ycombinator.com/item?id=18092837 - Sept 2018 (14 comments) Nuitka: A Python compiler - https://news.ycombinator.com/item?id=17683932 https://news.ycombinator.com/item?id=17683932 - Aug 2018 (6 comments) Nuitka: A Python compiler - https://news.ycombinator.com/item?id=16980704 https://news.ycombinator.com/item?id=16980704 - May 2018 (4 comments) Nuitka: a Python compiler - https://news.ycombinator.com/item?id=15354613 https://news.ycombinator.com/item?id=15354613 - Sept 2017 (60 comments) Nuitka Progress in 2015 – Python Compiler - https://news.ycombinator.com/item?id=10994267 https://news.ycombinator.com/item?id=10994267 - Jan 2016 (52 comments) Nuitka: a Python compiler - https://news.ycombinator.com/item?id=8771925 https://news.ycombinator.com/item?id=8771925 - Dec 2014 (135 comments) Nuitka — A Python Compiler - https://news.ycombinator.com/item?id=1746340 https://news.ycombinator.com/item?id=1746340 - Oct 2010 (33 comments)
codethief 5y ago
How does Nuitka deal with external resources (files) that the code pulls in via `open()`? Can I somehow include those in the executable if I want to?
silisili 5y ago
Big fan of Nuitka, and specifically the author Kay. I sent him an email years back, just thanking him for his work and asking how to donate. I was surprised when he wrote a very thoughtful and kind reply, acting rather shocked why someone would give him money. He's a real gem of a person, and I'm glad his work is getting (rightly) recognized.
malkia 5y ago
We have an internal tool that autoformats code (it calls clang for C++, some other tool for C#, and autopep8 for python). But for Python I did not want to rely on python, but have it a separate standalone .exe that can run by itself (almost, still needed python38.dll). So nuitka helped me there: I wrote a small app, which is simply: autopep8tool.py: import autopep8; autopep8.main() Then had this .bat file to do the .exe for me: autopep8tool.make_exe.cmd: @echo off pushd %~dp0 call "%VS140COMNTOOLS%\..\..\VC\bin\amd64\vcvars64.bat" p4 edit ..\tools\autopep8tool.\* p4 edit ..\tools\python38.dll for /d %%i in (*.dist *.build) do rd %%i /s /q echo import autopep8; autopep8.main() > autopep8tool.py call nuitka --standalone --assume-yes-for-downloads --unstripped --windows-dependency-tool=pefile autopep8tool.py copy autopep8tool.dist\autopep8tool.\* ..\tools copy autopep8tool.dist\python38.dll ..\tools for /d %%i in (*.dist *.build) do rd %%i /s /q popd works great so far, and even that the produced .exe is 10mb + 4mb python38.dll, it's great logistical saver (IMHO), as I can rotate the tool to other teams, depots, without requiring proper python (which is even trickier on windows). It's probably not completely sandboxed (hermetic), but works well so far. calling autopep8.exe --help even gives the right help. So great tool (though I've been told about others, like google's subpar, etc.)
- dataflow 5y ago
  Do you find it increases the maintenance burden? Since now a previously 1-character change requires a massive recompilation, and updates to Python (which happen frequently...) also require recompilation.
  - arthurcolle 5y ago
    you could just set up a file system listener and then trigger recompiles at some arbitrarily decided-upon interval of time. Doesn't seem like a huge pain otherwise.
  - malkia 5y ago
    I've only had to compile it once, unless we change python drastically and/or want autopep8.py changes (I would think this does not happen often, and if it does it also means we'll have to reformat our code). Granted, my case is really not general one, but nuitka fit well there (like I really wanted to avoid another 'freeze' tool that unpacks python to a temp folder, and calls it from there just to run autopep8.py). That to be said, I was not able to get nuitka working with some of the other popular format lib (I'll need to look again, it takes a bit of setup)
toolslive 5y ago
has it reached fixpoint?
codesnik 5y ago
is nuitka compilable with itself?
yummypaint 5y ago
Anyone have a sense of how this compares with cython performance wise?

502 points by stunt 5y ago

KaiserPro 5y ago

This looks rather fun, has anyone here used it in anger?

aogl 5y ago
I wrote about this over a year ago over here https://ao.ms/how-to-package-a-python-app-using-nuitka/ https://ao.ms/how-to-package-a-python-app-using-nuitka/ I've also used it in a production estate, and it works wonderfully well. Would recommend!
- KaiserPro 5y ago
  smashing, thanks for this.
rvieira 5y ago
I've used it for some simple, personal usage CLI tools and simple REST services. Works out-of-the-box and really well. From my experience some libraries are hard to compile (e.g. BeautifulSoup) but not impossible they need some tweaking. Sometimes I struggle with embedding data for some reason (especially since I'm spoiled with Go 1.16 `// embed`).

Yajirobe 5y ago

Nuitka is the best python compiler I've used. I have tried at least three others and none worked as good as nuitka. And I'm talking about complicated dependencies such as pytorch, selenium, tesseract - that type of stuff.

FartyMcFarter 5y ago
How do the resulting executables perform, compared to using CPython? Did you happen to run any benchmarks?
- avinashv 5y ago
  Not the OP, but it claims to be ~2x faster than CPython. I haven’t done extensive benchmarking, but for my small projects that seems about right.
  - dagw 5y ago
    Based on my own tests, once you have a 'normal' program that does a bit of IO, calls out some C based libraries etc. etc. I've rarely seen more than 10-20% performance increase across the whole program run. For individual functions you can see 2-4 times speed up over pure python. Most importantly, I've never seen it result in slower than cpython performance.
  - jhgb 5y ago
    That sounds significantly slower than PyPy on average, which would hardly make it "the best python compiler". YMMV, of course.
    - throwaway894345 5y ago
      It's all about compatibility to be honest. PyPy has done yeoman's work and it's still lacking in compatibility for important packages (e.g., psycopg2). Performance means little if we can't use major parts of the ecosystem.
      - jhgb 5y ago
        Is it preferable on PyPy to use a libpq wrapper over a native solution? As far as I can tell, even though CPython C API is in some way supported on PyPy it's really only advisable to use packages using it as a last resort for performance reasons. Aside from psycopg2cffi, which you should be able to use with PyPy, I'm pretty sure I saw other PEP 249 implementations for PostgreSQL the last time I checked (which admittedly was a few years ago).
        throwaway894345 5y ago
mimimi31 5y ago
>complicated dependencies such as pytorch, selenium, tesseract When I last tried Nuitka I couldn't get it to work with pycryptodome. Sounds like I should maybe give it another go.
- gota 5y ago
  I had a similar error. If I recall correctyl, I had to copy the contents of lib/Crypto into the nuitka dist folder, and then it worked.
neolog 5y ago
What the heck use case is this?

truncate 5y ago

This looks pretty cool. A better summary at the top would be nice to new users. I had to scroll down a bit, and read few lines until I realized it translates Python to C, and then uses CC to do final compilation (it is also quite ambiguous if executable is native or interpreted, until you read that part). I'd be looking more into this, but anyone else knows any other dynamically typed language which compiles to native code, not JITed or bytecode (I remember there were few Lisps out there).

astrange 5y ago
There are some for Perl in various degrees of emitting bytecode to actually optimizing (perlcc, RPerl). I think others prefer emitting to Java or making their own JITs.
scarygliders 5y ago
My one claim to fame is that a few years ago, I suggested the TL;DR wording on the Nuitka Overview page: https://nuitka.net/pages/overview.html https://nuitka.net/pages/overview.html If I could find the message I sent to the Nuitka mailing list (or did I suggest it via email? Can't remember) I'd publish that here. ;)
- scarygliders 5y ago
  ^ Seriously? Downvotes? What did I say which was wrong? Honestly thought it was a mildly interesting anecdote! Sheesh! You people! :)
cestith 5y ago
Steel Bank Common Lisp compiles Lisp to executables. Julia supports both a REPL and AOT compilation. There are some Forth compilers for various systems. https://www.thefreecountry.com/compilers/forth.shtml https://www.thefreecountry.com/compilers/forth.shtml
- eesmith 5y ago
  Humorously, https://cliki.net/Python https://cliki.net/Python - "Python is the name of a free high-performance compiler originally developed at Carnegie-Mellon University for CMU Common Lisp, it is now used by SBCL ..."

foepys 5y ago

If you are looking for something lighter: CPython can execute ZIPs with a Python program inside. You can also prepend the python shebang before the ZIP and mark it as executable in the filesystem. youtube-dl gets distributed like that.

randlet 5y ago
The problem with this method is that the user still needs to have Python on their system. Pyinstaller (Py2exe, cx_Freeze) bundle the python interpreter along with your app.
- londt8 5y ago
  I think its typical that only one python runtime is installed in the system. What is the use case for bundling the runtime with the app?
  - scarygliders 5y ago
    I package Captain's Log [0] with PyInstaller. I need to, because otherwise I wouldn't be able to package it for Windows users. The app is also closed source. I cannot easily package the application for Linux users, because of so many different flavours of Linux, so many different combinations of Python and Qt versions. Which is why I've given up on a pure Linux version and I'm in the middle of adapting it to work under WINE alongside Elite: Dangerous also running under WINE, instead. [0] https://captainslog.scarygliders.net/captains-log-2/ https://captainslog.scarygliders.net/captains-log-2/
  - nsomaru 5y ago
    Distribute my cli app + gooey as a stand-alone binary for windows users
  - curryst 5y ago
    Many systems still have both Python 2 and Python 3 installed. Also, in the standard case, the version of Python is controlled by the user, not by your app. Bundling it with the app lets you make sure that it's running Python 3.8, for example, so that you can use Python 3.8 features without having to add a bunch of compatibility shims in case the user is running an older version.

rlampert 5y ago

It is instructive to read GvR's take on Nuitka: https://news.ycombinator.com/item?id=8772124 https://news.ycombinator.com/item?id=8772124 GvR and the old boys currently present themselves as CoC proponents, and strategically use the CoC against anyone who criticizes them. Compare with his current Microsoft sponsored activities. If anyone would use that language against "his" (i.e. Mark Shannon's) project, they'd be out and suffer public defamation in no time.

wly_cdgr 5y ago
Wow, didn't realize he could be such a baby Edit: wow GvR's pals and fanboys are salty about someone pointing out what a loser he can be
gedazz 5y ago
What does CoC stand for?
- isp 5y ago
  CoC = "Code of Conduct" I haven't followed Python dev, so can't comment about how the CoC is being used in practice.
  - WesolyKubeczek 5y ago
    > I haven't followed Python dev, so can't comment about how the CoC is being used in practice. In tough situations, CoC powers are inversely proportional to the height of your position on the totem pole.
- Joker_vD 5y ago
  Still can't believe that this abbreviation is actually in use without any objections while there are talks about needing to rename the Coq language.
  - shpongled 5y ago
    I imagine that most people saying "CoC" out loud are not saying "cock", instead either "Code of conduct", or "cee oh cee".

myWindoonn 5y ago

I wish Nuitka were a drop-in solution, like PyPy, but it seems like I've never had a project that is 100% Nuitka-compatible.

ok123456 5y ago
Any scientific computing application that uses numpy, that is the vast majority of them, will not work with nuitka.
- gota 5y ago
  Hey, I did just that last week (actually I import Pandas, which requires numpy). There's a "plugin" option for numpy, in Nuitka, that makes it work. There are similar plugins for a handful of packages.
  - gota 5y ago
    For reference, the page with the plugins' documentation is not a very top search result, so I'll include it here: https://github.com/Nuitka/Nuitka/blob/develop/Standard-Plugins-Documentation.rst https://github.com/Nuitka/Nuitka/blob/develop/Standard-Plugi... The list of plugins is: dill-compat Required by the dill module eventlet Required by the eventlet package gevent Required by the gevent package multiprocessing Required by Python's multiprocessing module numpy Required for numpy, scipy, pandas, matplotlib, etc. pmw-freezer Required by the Pmw package pylint-warnings Support PyLint / PyDev linting source markers qt-plugins Required by the PyQt and PySide packages tensorflow Required by the tensorflow package tk-inter Required by Python's Tk modules torch Required by the torch / torchvision packages

gardnr 5y ago

Does this mean python is now self-hosted?

neolog 5y ago
Python is self-hosted under PyPy.

shivekkhurana 5y ago

I used Nuitka many years ago, good to see it on the front page.

BlankReg 5y ago

I've used Nuitka a couple times for some commercial Raspberry Pi projects and it worked perfectly each time.

Operyl 5y ago

Another cool project like this is https://github.com/Pypperoni/ https://github.com/Pypperoni/ but it takes a bit more of your own set up to get going. Disclaimer: just a happy user.

pansa2 5y ago

Is Nuitka able to create self-contained binaries that can be distributed as a single file? Or do they still have external dependencies such as the Python runtime? I’ve chosen Go rather than Python for a few small projects recently. Not for performance, parallelism or a particular fondness for the language, but just because Go can build truly standalone executables.

reacharavindh 5y ago
The last time I tried this was quite a while ago and your question was exactly why I tried. But, it failed with some unintuitive errors in dependencies of my Python script, and I gave up after 10 mins of Googling. I’m still wary of that experience and will avoid Python where I have such deployment needs unless the language natively comes up with such a build solution. FWIW, I think borg(backup solution written in Python and C) uses pyinstaller to get a single binary executable. It may be of interest to you.
thinker5555 5y ago
According to the readme: > The created binaries can be made executable independent of the Python installation, with --standalone and --onefile options.
vascocosta 5y ago
I use Pyinstaller to generate single file executables of my Python projects, which use Qt. It works great.
- XzAeRosho 5y ago
  The only problem I have found with Pyinstaller is that their "standalone" builds still depends on certain libraries that will change with the version of Python in the host OS, such as the glibc library, or others. The proposed solution by Pyinstaller is to build your program in the "oldest" version you support, which is silly compared to build systems like Go's, which are true standalone. Having said that, I'm curious about this solution, since it seems to claim a true standalone build...

jftuga 5y ago

How does this compare with PyInstaller?

woodrowbarlow 5y ago
pyinstaller doesn't compile your code. instead, it bundles a copy of the python interpreter along with your code, and sets everything up with an executable header so you can double-click the file to run your code within the embedded interpreter.

liveoneggs 5y ago

anyone tried GraalVM Python for this?

anentropic 5y ago
I looked into the this question for Graal Nodejs and I think it has the same problem in Python... Basically Graal Python and Nodejs each provide a custom interpreter for the target lang, the main goal of which is to provide interoperability with the Graal polyglot ecosystem. So you can run your python code under the GraalPython interpreter and it will run JITed fast and can import libs from other Graal-supported langs. But as far as outputting executable binaries, Graal only provides that for JVM projects and LLVM languages like C/C++/Rust. So it's not impossible, but you have to build your own Java wrapper project that loads the Graal Python interpreter class in code and then runs your python lib inside that. I expect eventually that boilerplate step can be automated as part of the Graal build tools. If I've got this wrong I welcome any corrections!
- anentropic 5y ago
  I was half-remembering what I found when researching Graal Nodejs clearer, less-garbled version as an answer here: https://stackoverflow.com/a/67331258/202168 https://stackoverflow.com/a/67331258/202168

Cadwhisker 5y ago

I've not tried Nuitka yet, but I've done similar things with 2 other tools to package up a Qt-based GUI tool that has a Windows installer, Mac ".dmg" or a Linux package: * FBS (FMan's Build System) - worked well for a while, but we couldn't upgrade Qt past a certain version and he version we used was only happy with Python 3.6. * Beeware - our current solution. Very, very happy with this; works smoothly, lets us use the latest Python and Qt versions and lets us make professional-looking installers that we can distribute.

mherrmann 5y ago
fbs author here; The Pro version of fbs [1] now supports Python > 3.6 and recent Qt versions. 1: https://build-system.fman.io https://build-system.fman.io
- diogenesjunior 5y ago
  What I love about HN. You can discuss almost anything and the project's creator pops up in the thread.
- Cadwhisker 5y ago
  Ah, great news. Thanks for making FBS; it is incredibly liberating to find a tool that lets us write Python GUI apps and have them appear as if they were made by XCode or Visual Studio.
  - mherrmann 5y ago
    Happy if you find it useful!
- sireat 5y ago
  Nice option for building Python+Qt applications! Buying the Pro version still leaves one to deal with Qt licensing does it not?
  - mherrmann 5y ago

brenainn 5y ago

I remember this being promising but struggling with a dependency that was using entry points to dynamically load plugins, and I couldn't get Nuitka to register that. Seeing this is a good reminder to have another shot at it.

jll29 5y ago

hello.py 1-liner on Ubuntu Linux: $ python -m pip install nuitka $ python3.8 -m nuitka hello.py install Nuitka with no problems and compiled "hello world", also with no problems. It created a 4,060,104 byte hello.bin with no symbol table, so no need to run strip to make it smaller, which again worked like a breeze. (The developers say on GutHub they will work on smarter ways to keep the dependency list small, which can at present escalate for modules like pandas that themselves have >1000 dependencies.)

zoom6628 5y ago

Have used on home projects couple of years ago to compile code on macOS and run on Windows and on Debian(on a pi). Worked well. Looking forward to retrying nuitka again once my current project gets to point I've finished the firmware and get back to working on middleware. Def worth a look for anybody that wants to distribute python developed system without installing python or the inevitable many dependencies of projects. It just makes distribution a whole lot more predictable. And to the reader who asked about GraalPython its on my todo list - have it installed but life, children, and work keep getting in the way.

jonnycomputer 5y ago

I suppose if my target is on x version of y OS then I'll have to compile on that machine, yeah? Sorry, I don't have much experience creating these sort of packages for distribution, but have recently found the need.

killingtime74 5y ago
It depends on the language. Could require it or could not. Not sure about here.

samsquire 5y ago

I love Python so much. It's my favourite language. It's so easy. I don't even use the dynamic nature of Python - monkey patching etc. I wish more people wrote algorithms in Python for understandability then compiled them into a compiled language for performance. I find that Python resembles pseudocode because it is syntactically sparse, unlike C++. In Python I don't need to worry about ownership or memory. It probably needs Pointers to be a low level system language. I just wish Python had parallelism.

divbzero 5y ago
By parallelism do you mean something different from multiprocessing? [1] [1]: https://docs.python.org/3/library/multiprocessing.html https://docs.python.org/3/library/multiprocessing.html
- triceratops 5y ago
  Pretty sure they're referring to the GIL.[1] 1. https://realpython.com/python-gil/ https://realpython.com/python-gil/
- samsquire 5y ago
  I want Thread to run in parallel. I've tried multiprocessing but I found it buggy for my use case. I tried to create a worker queue and submit items to worker threads with JoinableQueue but eventually the system deadlocks and no progress is made. I'm not sure what the problem is. It never finishes. It's a sentence correlator using multiprocessing - it generates correlations of words in your sentences. https://github.com/samsquire/notebook/blob/master/sentence-correlator.py https://github.com/samsquire/notebook/blob/master/sentence-c... https://github.com/samsquire/notebook/blob/master/workers.py https://github.com/samsquire/notebook/blob/master/workers.py Maybe somebody can catch what I'm doing wrong. edit: it may be because i've forgotten to run task_done on joinable queue. edit2: yes it was that! it works!
  - aw1621107 5y ago
    Out of curiosity, is using multiprocessing.Pool’s map/etc.() methods not an option? It’s a higher-level API which would remove the need to manually manage workers/a queue.

jokoon 5y ago

On their website: > Future > In the future Nuitka will be able to use type inferencing based on whole program analysis. It will apply that information in order to perform as many calculations as possible in C, using C native types, without accessing libpython. They already claim a 300% speed boost, but with those type inferencing things, it will probably go a lot of further... Can't wait for this project to mature. I guess compiling to WebAssembly should be trivial then?

codethief 5y ago
Speaking of speedup and program analysis, there's also mypyc[0] which relies on type annotations and also claims a performance boost: > The mypy project has been using mypyc to compile mypy since 2019, giving it a 4x performance boost over regular Python. Does anyone know how mypyc and Nuitka compare in practice? [0]: https://github.com/mypyc/mypyc https://github.com/mypyc/mypyc

cduzz 5y ago

What's the advantage of this over docker? Presumably this just bundles a python runtime and a copy of the dependencies and hides it all inside some sort of huge file?

ebrewste 5y ago
I have a project that pyinstaller can make into a 12 MB single file. Docker turns that same project into a ~400 MB container. This is not considering the Docker install footprint.

sscarduzio 5y ago

Amazing idea, is there anything like this for Javascript?

dang 5y ago

Past related threads: Nuitka 0.6.0 released - https://news.ycombinator.com/item?id=18092837 https://news.ycombinator.com/item?id=18092837 - Sept 2018 (14 comments) Nuitka: A Python compiler - https://news.ycombinator.com/item?id=17683932 https://news.ycombinator.com/item?id=17683932 - Aug 2018 (6 comments) Nuitka: A Python compiler - https://news.ycombinator.com/item?id=16980704 https://news.ycombinator.com/item?id=16980704 - May 2018 (4 comments) Nuitka: a Python compiler - https://news.ycombinator.com/item?id=15354613 https://news.ycombinator.com/item?id=15354613 - Sept 2017 (60 comments) Nuitka Progress in 2015 – Python Compiler - https://news.ycombinator.com/item?id=10994267 https://news.ycombinator.com/item?id=10994267 - Jan 2016 (52 comments) Nuitka: a Python compiler - https://news.ycombinator.com/item?id=8771925 https://news.ycombinator.com/item?id=8771925 - Dec 2014 (135 comments) Nuitka — A Python Compiler - https://news.ycombinator.com/item?id=1746340 https://news.ycombinator.com/item?id=1746340 - Oct 2010 (33 comments)

codethief 5y ago

How does Nuitka deal with external resources (files) that the code pulls in via `open()`? Can I somehow include those in the executable if I want to?

silisili 5y ago

Big fan of Nuitka, and specifically the author Kay. I sent him an email years back, just thanking him for his work and asking how to donate. I was surprised when he wrote a very thoughtful and kind reply, acting rather shocked why someone would give him money. He's a real gem of a person, and I'm glad his work is getting (rightly) recognized.

malkia 5y ago

We have an internal tool that autoformats code (it calls clang for C++, some other tool for C#, and autopep8 for python). But for Python I did not want to rely on python, but have it a separate standalone .exe that can run by itself (almost, still needed python38.dll). So nuitka helped me there: I wrote a small app, which is simply: autopep8tool.py: import autopep8; autopep8.main() Then had this .bat file to do the .exe for me: autopep8tool.make_exe.cmd: @echo off pushd %~dp0 call "%VS140COMNTOOLS%\..\..\VC\bin\amd64\vcvars64.bat" p4 edit ..\tools\autopep8tool.\* p4 edit ..\tools\python38.dll for /d %%i in (*.dist *.build) do rd %%i /s /q echo import autopep8; autopep8.main() > autopep8tool.py call nuitka --standalone --assume-yes-for-downloads --unstripped --windows-dependency-tool=pefile autopep8tool.py copy autopep8tool.dist\autopep8tool.\* ..\tools copy autopep8tool.dist\python38.dll ..\tools for /d %%i in (*.dist *.build) do rd %%i /s /q popd works great so far, and even that the produced .exe is 10mb + 4mb python38.dll, it's great logistical saver (IMHO), as I can rotate the tool to other teams, depots, without requiring proper python (which is even trickier on windows). It's probably not completely sandboxed (hermetic), but works well so far. calling autopep8.exe --help even gives the right help. So great tool (though I've been told about others, like google's subpar, etc.)

dataflow 5y ago
Do you find it increases the maintenance burden? Since now a previously 1-character change requires a massive recompilation, and updates to Python (which happen frequently...) also require recompilation.
- arthurcolle 5y ago
  you could just set up a file system listener and then trigger recompiles at some arbitrarily decided-upon interval of time. Doesn't seem like a huge pain otherwise.
- malkia 5y ago
  I've only had to compile it once, unless we change python drastically and/or want autopep8.py changes (I would think this does not happen often, and if it does it also means we'll have to reformat our code). Granted, my case is really not general one, but nuitka fit well there (like I really wanted to avoid another 'freeze' tool that unpacks python to a temp folder, and calls it from there just to run autopep8.py). That to be said, I was not able to get nuitka working with some of the other popular format lib (I'll need to look again, it takes a bit of setup)

toolslive 5y ago

has it reached fixpoint?

codesnik 5y ago

is nuitka compilable with itself?

yummypaint 5y ago

Anyone have a sense of how this compares with cython performance wise?