4 ms·
A proof-of-concept Python executable built on Cosmopolitan Libc (2021)
- FazXTD 3y ago[dead]
- Retr0id 3y agoBased on the screen recording demo it looks unexpectedly slow. What's the bottleneck?
- BlueUmarell 3y agoThe "Python", the "Actually" and the "Portable" parts.
- Retr0id 3y agoI'm aware of both Python and Cosmopolitan implementation details and neither of these things explain a multi-second page load.
- bluetomcat 3y agoThe interpreter is linked to a cross-platform libc-like library, and compiled to a funky executable format, supporting multiple OS targets simultaneously. The initialisation and startup are probably computationally-intensive.
- Retr0id 3y agoWe can see the initial startup speed from the time it takes the command in the terminal to print something, which isn't so bad (looks like under a second). Maybe Flask is forking on each request, which I could imagine being slow, but not slower than the initial process startup.
- jart 3y agoahgamut is a college student. He doesn't have $15,000 workstations like us.
- jart 3y agoSoftware linked with Cosmopolitan Libc usually goes 2x faster than Musl. - https://justine.lol/cosmo3/ https://justine.lol/cosmo3/ - https://twitter.com/JustineTunney/status/1726141024597324189/photo/1 https://twitter.com/JustineTunney/status/1726141024597324189... What exactly looked slow to you?
- ahgamut 3y agoMy 4-core computer's probably building Cosmopolitan Libc in the background + the Python binary is 2.7 unoptimized. This blog post is originally from 2021.
- llm_nerd 3y agoFor those wondering, the submission is a library / tooling usage that generates a single binary that executes across Linux, Mac, Windows, and BSDs. The author defines that as Actually Portable (which was the confusing part to me as we all know that python runs on just about everything)
- Retr0id 3y agofwiw the definition doesn't come from the author, it's how Cosmopolitan/APE describes itself (APE being Actually Portable Executable - as opposed to a mere Portable Executable, aka EXE)
- thesuperbigfrog 3y agoJustine's αcτµαlly pδrταblε εxεcµταblε website: https://justine.lol/ape.html https://justine.lol/ape.html
- moffkalast 3y agoCouldn't the same result be achieved by just copying an entire virtualenv? Though that is rather annoying to source every time and in every terminal.
- salviati 3y agoAny binary dependency you have would need to be present for all target architectures in your virtualenv for this to work. But when you pip install a package only the binary for your current platform gets installed.
- moffkalast 3y agoI presume you'd have the same problem with the binary, there'd have to be separate builds for arm, x64, etc.
- ingenieroariel 3y ago
- idoubtit 3y agoBy "Actually Portable", this article means that one can build a portable executable of `python` (compiler and runtime) which can run on many systems, including Windows and Linux. But the reality is a bit different . The article states that their result is slow and passes one third of the regression tests. So it's a proof of concept, not a portable executable one can use for real-world projects.
- ingenieroariel 3y agoThe reality is a bit different, the work on Python 3.6 was checked into the Cosmopolitan repo and I have been able to use it for production workloads that are in pure python. [0] As Cosmopolitan Libc has evolved, it has been possible to compile more software without modifications, and that includes latest Python through a project called superconfigure[1]. Last person who tried to reproduce it from scratch did it last week (granted it too them a few days of solid work) but in the end they ended with a portable binary with Python 3.11.9, brotli, ssl and asyncio for their work related project.[2] [0] https://github.com/jart/cosmopolitan/tree/master/third_party/python https://github.com/jart/cosmopolitan/tree/master/third_party... [1] https://github.com/ahgamut/superconfigure/ https://github.com/ahgamut/superconfigure/ [2] https://github.com/croqaz/cpython/ https://github.com/croqaz/cpython/
- pierrebai 3y agoI find the use case where you have a system in production but could not possibly use the official Python for that system... unusual? Double so since Python code (.py) is itself portable, and I would assume that the amount of work to make all the necessary extension be built into your APE vs just using the standard pip/pipenv/poetry/etc to be in favor of the latter. Not even talking about the maintainability of the resulting system by other people who'd have to learn how to handle building an APE.
- ingenieroariel 3y agoThe use case was a client creating electrification plans based on structures from satellite data. We were able to get rid of gdal/pandas/networkx and several other dependencies and ended up with a fast python based process that could be given to clients for them to reproduce on their own machines (windows workstations). In my use case, the niche is not having WSL/Docker available and letting end users repeat studies or re-run configuration scripts.
- JamesCoyne 3y agoThis is a good place to mention https://nuitka.net/ https://nuitka.net/ which aims to compile python programs into standalone binaries.
- ijustlovemath 3y agoI've had better luck with pyinstaller! Nuitka is buggy when you use multiprocessing
- cmehdy 3y agoSame here. Wrote a simple program to which I added Fire to parse arguments as a CLI, and Gooey/Tkinter for a GUI on top of it. To get a working standalone .exe file which does not require folders on the side, Pyinstaller did the job. Unfortunately it also triggers antivirus scans for some AVs..
- dboreham 3y agoNot binaries, but I've had success with Shiv[1] which builds a Python application into a single-file package that can be run on any machine provided it has the Python binary installed (but not much else). We use it to ship products that run as-is on both Linux (including WSL2) and macos. [1] https://shiv.readthedocs.io/en/latest/ https://shiv.readthedocs.io/en/latest/
- ahgamut 3y agoI've tried nuitka before, and a recent question that occurred to me was: does nuitka have an option to output just C files? Something like: python -m nuitka example.py --no-compile Might be interesting to see if the above is possible. We could get things like a completely-statically-compiled Python stdlib within the APE.
- tripflag 3y agoI built a Win10 binary with Nuitka just the other day, and was surprised to find the pyinstaller binary had higher performance at runtime. Pyinstaller also had several other advantages, such as producing smaller binaries, and building faster, and Win7 support. For reference, I kept notes on the exact commands I used: https://github.com/9001/copyparty/blob/hovudstraum/docs/nuitka.txt https://github.com/9001/copyparty/blob/hovudstraum/docs/nuit...
- FloatArtifact 3y agohttps://gregoryszorc.com/docs/python-build-standalone/main/ https://gregoryszorc.com/docs/python-build-standalone/main/
- FrustratedMonky 3y agoThe article's ambitious promise of an "Actually Portable" Python executable across diverse systems such as Windows and Linux ends up revealing a rather melancholic truth. What we encounter instead is a slow, underperforming proof of concept that passes merely a third of its tests. Is this not, in some perverse way, reflective of our contemporary technological striving, where the ideal of universality often crumbles under the weight of practical limitations? Are we not, in our pursuit of perfect tooling, always chasing a mirage? The reality, starkly exposed by these tools, might be that our pursuit is as flawed and fragmented as the tools themselves.
- kitd 3y agoThe good news is that it's 2024, this article is a few years old and many of the problems mentioned have been (or are being) ironed out.
- m0shen 3y agoThis article was written when cosmopolitan: https://justine.lol/cosmopolitan/index.html https://justine.lol/cosmopolitan/index.html was v2. v3 has since been released: https://justine.lol/cosmo3/ https://justine.lol/cosmo3/ . https://cosmo.zip/ https://cosmo.zip/ Also contains a python build https://cosmo.zip/pub/cosmos/bin/python https://cosmo.zip/pub/cosmos/bin/python , though I'm not sure if the problems mentioned in the article are solved.
- ahgamut 3y agoHey, this is my post from 2021, when I was testing Python2.7 and Python3.6 with Cosmopolitan Libc on an old 4-thread Haswell. It's now a lot easier to build Python (and gcc, gnu coreutils, curl etc.), and the binaries are faster, multi-threaded, and quite convenient to use. There are lots of interesting directions to explore when it comes to building software with Cosmopolitan Libc.
- ingenieroariel 3y agoThanks for updating the blog post too with the datasette screen recording, the speed difference is quite noticeable. Adding it here since a few people were wondering about it in the comments, but feel free to check the original article for the 2024 update: https://ahgamut.github.io/images/ape-datasette.gif https://ahgamut.github.io/images/ape-datasette.gif
- AutoArchiverBot 3y ago[flagged]
- johnea 3y agoHaHa! 2 weeks too late for April Fools...
- dang 3y agoRelated: Python is Actually Portable - https://news.ycombinator.com/item?id=32245430 https://news.ycombinator.com/item?id=32245430 - July 2022 (120 comments)