7 ms·
Nuitka: a Python compiler
- dang 9y agoPrevious discussions: https://hn.algolia.com/?query=Nuitka%20points%3E3&sort=byDate&dateRange=all&type=story&storyText=false&prefix&page=0 https://hn.algolia.com/?query=Nuitka%20points%3E3&sort=byDat....
- vog 9y agoHow does this compare to PyPy in terms of speed and optimizations?
- hamilyon2 9y agoIt is generally slower and much more predictable and regular
- tsavola 9y agoWhich one?
- chrisseaton 9y agoI would imagine Nuitka is slower, because it cannot apply dynamic optimisations which are the key technique for making a language like Python fast, but more predictable and regular as the dynamic optimisations that PyPy uses are vulnerable to performance cliffs.
- radarsat1 9y agoAre there any profile-guided optimisation methods for Python? Like, something that saves information from PyPy's dynamic optimisations on first run and uses them later for an AOT approach?
- stevedonovan 9y agoYes, compare to LuaJIT. On a good day, it can bite C's shiny metal ass, but consistent and reliable application performance requires intimate knowledge of those performance cliffs.
- Sean1708 9y ago> it can bite C's shiny metal ass Can it actually? I've certainly seen benchmark's where a JITed program is comparable/marginally faster than a C program, but I've never seen one where it outright trounces C.
- xapata 9y agoYes. A jit compiler can compile based on both code and data, and can compile all the dependencies. In contrast an aot compiler must only use the code and links to other pre-compiled binaries. Further, a jit can compile to specific hardware. An aot must anticipate what hardware is possible. If you aot compile to several possible hardware configs, then you'll bloat the binary.
- MaxBarraclough 9y ago> In contrast an aot compiler must only use the code Not since the development of "profile-guided optimisation", no.
- xapata 9y agoOk, then you've got the code and the profile. The actual data and hardware it happens to be using is still better information.
- MaxBarraclough 9y ago> The actual data and hardware it happens to be using is still better information. That's certainly what HotSpot's promoters want us to believe, but I'm not convinced there's a lot of extra value in having detailed data on the current run, compared to 'static' profile-guided optimisation. I'm not sure if there's been any serious study of this question. As to the specific hardware: this might matter for things like SIMD, yes, but often won't make any appreciable difference. Some Linux distros have the user compile almost everything locally, but aren't all that much faster for it.
- DonbunEf7 9y agoCompared to PyPy, Nuitka is slow, buggy, and doesn't handle as much of the Python language. Nuitka vomits and dies on about half the code I've tried it on.
- hultner 9y agoI tried this one a couple of years ago on two existing project (one fairly new product in development and one mature product) after some irritation with pyinstaller, etc. But ran into compatibility problems with most libraries we used (one application were a native QT-based and one were for web based on Falcon). Not sure how it looks today but if it's anything like before I'd recommend developing with it in mind from scratch if you want to use it.
- baldfat 9y agoHaven't peeked at using python since making portable executable programs was just so frustrating. How is Python now compared to a few years ago?
- edraferi 9y agoThe Talk Python To Me podcast just did an episode on this [0] It's mainly an interview with Glyph (author of Twisted) about his PyCon 2016 talk "Shipping Software To Users With Python" [1] [0] https://talkpython.fm/episodes/show/127/shipping-software-to-users https://talkpython.fm/episodes/show/127/shipping-software-to... [1] https://www.youtube.com/watch?v=5BqAeN-F9Qs https://www.youtube.com/watch?v=5BqAeN-F9Qs
- baldfat 9y agoThanks. Good talk. He does say my favorite thing about Python. Python is the second best language for anything. I use R for stats now because that is mostly what I use for work and I use Racket because I can punch out a bug free easy to compile program in a short afternoon for everything else right now.
- baldfat 9y agoAfter listening I have less hope this ever gets fixed.
- falcolas 9y ago> making portable executable programs was just so frustrating Hasn't changed much, but Docker has made many of the distribution problems more-or-less irrelevant (but only if you use Docker).
- rplnt 9y agoIf performance is your only motivation, I recommend you have a look at cython (previously pyrex). It basically makes writing C modules a breeze for a python programmer because you write them in type annotated python. Regular python is transpiled into mess of python calls, your typed tight loop will be "raw" C.
- dec0dedab0de 9y agocython (previously pyrex) Ive never used either outside of a toy implementation, but Cython is based on Pyrex. They are different projects.
- 3pt14159 9y agoI've been using cython for years, it is indeed great. I don't recommend anyone write multi-platform raw C any more unless you have a really good reason to. You can even make C-extensions for things like Ruby! But I have a new project now that needs to be both highly concurrent as well as reasonably fast and I'm considering giving Numba a try since it seems like it's both faster and easier to work with. Does anyone with experience with both have a comment as to the downsides of Numba vs Cython?
- jacobush 9y agoIs it easy to compile standalone executables with Cython? Or do you always have to drag around the full(-ish) Python environment? For me, shedskin was the sweet spot on paper, unfortunately it doesn't work very well - I can never get its output to compile properly. Also it's Python 2.x only.
- dagw 9y agohttps://github.com/cython/cython/wiki/EmbeddingCython https://github.com/cython/cython/wiki/EmbeddingCython
- syllogism 9y agoIf you're trying to rescue existing code with type annotations, Numba can be a good solution. I think Cython has a better build story for shipping libraries though. I think users need Numba if you use it -- that's not true with Cython. If you're writing fresh code, I think Cython is much better. Numba doesn't really give you a language to work in. With Cython you can basically write C or C++ --- you can declare structs, manage pointers, even have C++ classes with templates if you want. This hash table does similar work to Google's dense_hash_map, with similar performance: https://github.com/explosion/preshed/tree/master/preshed https://github.com/explosion/preshed/tree/master/preshed . I think it's hard to write something like this with Numba.
- qaq 9y agoif you want to try a compiled lang. with Python like syntax and speed comparable to C I would strongly recommend Nim.
- lima 9y agoI tested this with a large project that uses asyncio, Tornado and RabbitMQ and it, surprisingly, worked. Took half an hour to compile though.
- sametmax 9y agoNuikta is very much magic to me. The author is alone, making steady progresses on a crazy complicated project, always communicating regularly, in an humble tone. Then you download the stuff, and it works. I'm still amazed on things like this happening. I though those events stopped existing after the 2000 bubble.
- ctoth 9y agoI completely agree with your comment. It seems like most people here are bashing on this in one way or another. People talking about Cython, people talking about how they don't even use Python anymore, etc. etc. This makes me super sad, because this is seriously one of the most technically amazing projects I have seen in several years. It is a labor of love by one single guy who frequently posts status updates where he's accomplished some new thing that was thought to be impossible. It's really an inspiring story of somebody who isn't stopped by the haters, building something that he wants, that lots of people have told him is impossible or useless, but yet he still continues to make steady progress plugging away at it. I really used to identify with the average commenter on this site, and feel that there was really a time when something like this would be truly understood and appreciated... But now, rather than some Python expert explaining how incredible this is, or some compiler wizard spotting this, being interested and offering to help we have lots of sniping from uninformed people who just seem to be annoyed that this won't magically speed up their Python now Can we have a technical deep-dive on what this guy's accomplished? A few clever curious hackers tearing about the code spotting all the brilliance and sharing bits of it with us with commentary? No. Let's just all go on and on about how we should just rewrite everything in Rust. Again.
- sametmax 9y agoBTW, I donated to the project, and I invite everybody with any remote interest in compiled Python to do so. It goes a long way, and it's easier than contributing if you have no C++ background.
- jokoon 9y agoI wish there was as much money spent on python as there is for javascript.
- toyg 9y agoI often think the same, then some well-intentioned corporate drone comes along trying to reinvent java WARs in Python and I feel so nostalgic of when I was the only one in my office who knew what Python was.
- zitterbewegung 9y agoIsn't making Python easy to deploy a pain point for people though? It is for me.
- toyg 9y agoDepends what you mean by “deploy”. Some use cases are very simple. As always, it’s about trade-offs.
- xapata 9y agoWhat specifically is difficult? The folks at Anaconda can probably help you. Use the conda package manager.
- takeda 9y agoIt is easy, just use setuptools and create your own setup.py then ./setup bdist_wheel and you have a single easy to deploy (pip install <package>, or put in an internal PyPi repo) package. I especially encourage to use virtualenv too keep your application completely disjoint from the rest of your system. It really helps when in the future you decide to upgrade the OS.
- kalefranz 9y agovirtualenv keeps pure-python code isolated from other python code associated with that interpreter. Any python code that makes use of c libraries and other system libraries is usually not isolated. In fact it can be quite leaky and cause all sorts of difficult to identify problems.
- indubitably 9y agoI don't understand why Python people aren't more intent on getting it into web browsers. Or are they?
- Derbasti 9y agoWhat would I want to do with Python in a web browser? I wouldn't be able to use most of the standard library, and I wouldn't be able to use most PyPi packages. Sure, it might be a better language than JavaScript, but you would have to rip out and replace the entire standard library, and Python without its libraries wouldn't be any fun.
- sametmax 9y agoOh we wish. People tried (brython came close). But any attempt are currently not practical as the runtime is very heavy, not to mention the stdlib. So we are stuck unless the browser vendors decide to ship a Python VM next to their JS VM. And they won't do that.
- fulafel 9y agoClojureScript does very well despite the parent Clojure having a big runtime and largish stdlib. Good tree shaking (dead code elimination) helps a lot.
- paulddraper 9y agoSpecifically, tree shaking through the venarable (and somewhat confusingly named :) Google Closure Compiler.
- Splendor 9y agoIs there a simple example somewhere of the types of optimizations Nuitka makes? I clicked around the site and couldn't find one.
- temeritatis 9y agoWorks well with pyqt as well. It's a dream environment writing cross platform apps, and since they're compiled, distribution (especially on win/osx) is really simple. Highly recommended!