8 ms·

Pex: A tool for generating .pex (Python EXecutable) files, lock files and venvs

138 points by eamag 2y ago

MikeTheGreat 2y ago
I'm always on the lookout for ways of packaging Python programs/script-piles into single-file executables, so thank you for posting this! The GitHub README says it builds on the Python .ZIP App format (PEP441), which in turns says you can put your app in a .ZIP file and Python will run it as an app. I think I went this route with PyInstaller's one-file output option. But I found it to be too slow because (1) my app was a CLI app and I needed it to start, run, and end quickly, (2) I imported something from Django because it was useful (which ballooned the size), and (3) the single-file .ZIP needed to be extracted into a temporary directory, run, and then deleted every time (!!!). Does anyone know how Pex / PEP441 deals with this? Is the extract-run-delete thing normal/standard? Is there a way to cache the extracted app?
- tylerhou 2y ago
  My guess is that without operating system support or equivalent, there is not an easy way to avoid extracting the zip file. (You could patch syscalls in the Python interpreter to read from the zip file for certain paths, but that might be hacky.) I did find https://github.com/google/mount-zip https://github.com/google/mount-zip which might be useful, but you would likely have to still mount the zip manually. However, you don’t have to worry as much about cleaning up after yourself.
- redleader55 2y ago
  You might find interesting looking at XAR[0] which works by mounting a squashfs filesystem of the "archive", instead of using a zip file. The squashfs is mounted/unmounted "lazily" meaning if you run it a few times in a row it will only mount it one time. [0] -https://github.com/facebookincubator/xar https://github.com/facebookincubator/xar
glass-z13 2y ago
I used python to write a small utility that i wanted to share with a coworker, ended up having to rewrite it in Go because i spent 3 hours trying to compile the python version to an exe so that it was easier to run on their machine. I wonder why isn't there an easy way to just package the interpreter inside an EXE for people that don't care about binary size just so that it would make it seamless to package up utilities written in python.
- Esgrove 2y ago
  There is: PyInstaller
  - glass-z13 2y ago
    Seems like i have it in my browsing history, i remember not being able to run the executable it produced, not being able to find a library it was supposed to have. This was just using the default "pyinstaller your_program.py", and i was frustrated enough to not go deeper into why that was. Will definitely give it a try again in the future
    - randlet 2y ago
      Pyinstaller can be a bit fiddly to get right initially if you have a lot of package dependencies, but other than that it works really well.
      - Numerlor 2y ago
        Using the spec files for persistent readable configuration also goes a long way, if you treat pyinstaller as a python module you can automate it whole with just python, including the spec files as it executes them as python scripts I've had automated builds running mostly untouched for years here https://github.com/Numerlor/Auto_Neutron/blob/master/pyinstaller_build https://github.com/Numerlor/Auto_Neutron/blob/master/pyinsta..., though it doesn't have any particularly troublesome packages other than PySide6
    - WhyCause 2y ago
      PyInstaller-made executables also used to have a habit of getting flagged by security software as malicious (maybe that's why you couldn't run it?) -- apparently, so many malware writers used it that it ruined the party for everyone. Fortunately, that was only the 32-bit version of Python 2.7. Using 64-bit versions or Python 3 was enough to not get flagged as malicious. I figured that out when I decided I didn't want to teach myself Go just then to deploy something that had worked the day before.
BiteCode_dev 2y ago
I know it's not very nice to post that here, but given it's hard to get good info about python because the ecosystem is so huge, I'd rather give it straight: Don't use pex, use shiv: https://shiv.readthedocs.io/ https://shiv.readthedocs.io/ Pex doesn't work on windows, is slower than shiv and shiv solves the resource files problem that you will encounter with dependencies like django. To achieve this, shiv unzips the whole zipapp in a cache on disk and create a transparent proxy to it. Still, I wish zipapps had better tooling honestly. Using either pex or shiv is more complex than it needs to be. I hope uv will eventually give you something like "uv zipapp" and make it good and simple. Right now, to bundle all deps for shiv you have to download first all wheels, then figure out the proper incantation to include them. And if you use WSGI, you probably want some kind of dedicated entry point. The tech is cool, but it deserves maturing to be a true ".war for python".
- fsloth 2y ago
  ”Pex doesn't work on windows” Oh that’s such a weird limitation. In the early 2000’s open source hubris, next year was always the year of the Linux desktop. Since then consumer windows matured into a totally ok OS and one with the best support for graphics and C++ development at that. Non-windows support nowadays is a fairly strong signal of a non-serious software offering if there is no obvious reason for it. And that’s totally fine, hobby tools developed by enthusiasts rock - but they are not industrial in scope as such.
  - sneed_chucker 2y ago
    A lot of serious software offerings are only concerned with the server use case, modern servers run Linux unless there's a good reason not to, and modern windows has more than one acceptable way to run Linux binaries if you absolutely have to.
    - Vegenoid 2y ago
      > modern servers run Linux unless there's a good reason not to I think many people on HN would be surprised at how many orgs are using Windows servers heavily, because of their familiarity and comfort with Microsoft, or because some application requires it. Of course they are non-tech companies using the servers internally for enterprise applications, not web servers, but there is absolutely a lot of windows server usage in corporate environments.
serjester 2y ago
I think the easiest way to do this now is with uv and their support for inline metadata. At this point you can just have the user install uv with a single command and then have the user run the script with `uv run example.py`.
- d0mine 2y ago
  —script appears to be missing if you meant https://peps.python.org/pep-0723/ https://peps.python.org/pep-0723/
khaledh 2y ago
We've been using pex for a long while to package PySpark jobs with dependencies into a single file. Saves a ton of time vs. the old way of building/deploying docker images. https://spark.apache.org/docs/latest/api/python/user_guide/python_packaging.html#using-pex https://spark.apache.org/docs/latest/api/python/user_guide/p...
- ttyprintk 2y ago
  We shiv all the dependencies into a single file, bake it into a Docker layer, and refer to it in each scripts shebang.
- fire_lake 2y ago
  I would prefer Docker images since you they are easier to “cross compile”. However EMR requires the same Docker image for the entire cluster - why not per job!?!?
  - khaledh 2y ago
    That's fair. Pex does support specifying a target platform though, which can package linux binary dependencies on a Mac for example. That being said, if a native dependency doesn't provide binaries (i.e. source only) we're out of luck. We solve this issue by precompiling and publishing those dependencies on a private package index. Thankfully there are only a handful of dependencies that need this.
    - fire_lake 2y ago
      Unfortunately it’s the most important dependencies that tend to be native - Spark, TensorFlow, Torch, etc…
gavmor 2y ago
I can see how .pex or shiv's zippaps live at the Pareto threshold where converting a Python app to an 80% (or less...) executable takes only 20% (or less!) of the effort that would be spent re-writing the app in eg C, C++, Rust, or Golang. Still, it breaks my heart to see all this work being done to keep developers employed as "Python developers", instead of leveling them up to a broader programming skillset. It seems as if the industry is investing rather heavily in a local maximum. But, of course, the market is has information that I don't.
- Numerlor 2y ago
  Eh I wouldn't call PEX proper executables that'd replace the languages above as they still need python, but I still prefer python for things like simple Qt apps even if it means using e.g. PyInstaller
- slotrans 2y ago
  It is often the case that the nifty Python thing you want to pass around uses one or more nifty Python libraries that have no C/C++/Rust/Golang equivalent (or no obvious equivalent), and so rewriting it becomes a herculean task.
  - gavmor 2y ago
    I wonder if this isn't reflective of some universal law of niftiness.
- zem 2y ago
  > Still, it breaks my heart to see all this work being done to keep developers employed as "Python developers", instead of leveling them up to a broader programming skillset. It seems as if the industry is investing rather heavily in a local maximum. this is a topic near to my heart; I have spent the last several years building my career around doing this sort of work (specifically writing tools to expand the capabilities of python code). the key idea is not that we "keep people employed as python developers", it is that python (you can also substitute ruby, javascript, php, or any other popular dynamic language) is a genuinely powerful and expressive language from the point of view of converting ideas to code, and having that code be readable, maintainable, and extensible by other humans. where is falls down is on the "machine facing" end - you don't get the guarantees of static typing, the import system is a bit of a mess, it isn't as fast as lower level, statically-compiled-to-native-code languages, and (as the current discussion shows) packaging and distributing an end user binary is somewhere between a morass and a nightmare. however, none of these things are a problem with the language per se, and they can be fixed with some combination of improving the implementation, writing better tooling (cf valgrind from the c world), making it easier to write small bits of critical code in c/rust/zig/etc, and improving language features while not sacrificing what draws people to use python in the first place. is python a local maximum? almost certainly; it would be foolish to think that nothing better could come along, and indeed pyret already looks distinctly better in terms of pure dynamic language design. but the answer to that is not to "level up" people who are happy and productive using python to some language like rust or c++ that imposes a heavier cognitive burden on the user to take care of low level details that their applications might not need, or one like golang that pushes the complexity out into the application where problems need to be solved via design patterns that people need to learn and copy, rather than just building them into the language or stdlib.
mappu 2y ago
Pex and Shiv both require the end user to have Python installed, so I think PyOxidizer is the best tool in this category: https://pyoxidizer.readthedocs.io/en/stable/pyoxidizer_comparisons.html https://pyoxidizer.readthedocs.io/en/stable/pyoxidizer_compa...
- martsa1 2y ago
  How does Pyoxidizer compare to pyinstaller and nuitka?
  - mappu 2y ago
    Great question! That was actually covered in the link I posted. Compared to PyInstaller, it doesn't dump any temporary files on disk. Compared to Nuitka, it uses the real CPython, not a different implementation.
- stuhood 2y ago
  Not anymore: PEX now supports including an interpreter in the package: https://docs.pex-tool.org/scie.html https://docs.pex-tool.org/scie.html And unfortunately, PyOxidizer went into maintenance mode: https://gregoryszorc.com/blog/2024/03/17/my-shifting-open-source-priorities/ https://gregoryszorc.com/blog/2024/03/17/my-shifting-open-so...
- delijati 2y ago
  Adding the interpreter is fairly easy. I had in project to do it for windows. If someone is interested. https://github.com/delijati/docker-shiv?tab=readme-ov-file#add-python-interpreter https://github.com/delijati/docker-shiv?tab=readme-ov-file#a...
smallmouth 2y ago
I thought this was going to be a post about plumbing and then I read further and was even more excited to say the phrase "python executable". Very interesting. Thanks!
jbritton 2y ago
I use cx_Freeze to get an exe and Inno Setup for a Windows installer. It has worked well for me. I also have a DigiCert dongle to sign the exes.
NeutralForest 2y ago
I've quickly skimmed the comments and haven't found a mention of briefcase (https://beeware.org/project/projects/tools/briefcase/ https://beeware.org/project/projects/tools/briefcase/) which is a tool to build standalone apps in Python.
bborud 2y ago
The comments remind me of the Judean People’s Front scene from Life of Brian. Everyone has their solution to the problem - so there is no solution.
xrd 2y ago
I just put everything inside a docker container. But those 9GB containers when you use pytorch and Nvidia cuda libraries. Has anyone tried using this for that type of packaging?
shalabhc 2y ago
We have been using pex to deploy code to running docker containers in ECS to avoid the cold start delay. Cuts down the iteration loop time for development significantly. https://dagster.io/blog/fast-deploys-with-pex-and-docker https://dagster.io/blog/fast-deploys-with-pex-and-docker

138 points by eamag 2y ago

I'm always on the lookout for ways of packaging Python programs/script-piles into single-file executables, so thank you for posting this! The GitHub README says it builds on the Python .ZIP App format (PEP441), which in turns says you can put your app in a .ZIP file and Python will run it as an app. I think I went this route with PyInstaller's one-file output option. But I found it to be too slow because (1) my app was a CLI app and I needed it to start, run, and end quickly, (2) I imported something from Django because it was useful (which ballooned the size), and (3) the single-file .ZIP needed to be extracted into a temporary directory, run, and then deleted every time (!!!). Does anyone know how Pex / PEP441 deals with this? Is the extract-run-delete thing normal/standard? Is there a way to cache the extracted app?

tylerhou 2y ago
My guess is that without operating system support or equivalent, there is not an easy way to avoid extracting the zip file. (You could patch syscalls in the Python interpreter to read from the zip file for certain paths, but that might be hacky.) I did find https://github.com/google/mount-zip https://github.com/google/mount-zip which might be useful, but you would likely have to still mount the zip manually. However, you don’t have to worry as much about cleaning up after yourself.
redleader55 2y ago
You might find interesting looking at XAR[0] which works by mounting a squashfs filesystem of the "archive", instead of using a zip file. The squashfs is mounted/unmounted "lazily" meaning if you run it a few times in a row it will only mount it one time. [0] -https://github.com/facebookincubator/xar https://github.com/facebookincubator/xar

glass-z13 2y ago

I used python to write a small utility that i wanted to share with a coworker, ended up having to rewrite it in Go because i spent 3 hours trying to compile the python version to an exe so that it was easier to run on their machine. I wonder why isn't there an easy way to just package the interpreter inside an EXE for people that don't care about binary size just so that it would make it seamless to package up utilities written in python.

Esgrove 2y ago
There is: PyInstaller
- glass-z13 2y ago
  Seems like i have it in my browsing history, i remember not being able to run the executable it produced, not being able to find a library it was supposed to have. This was just using the default "pyinstaller your_program.py", and i was frustrated enough to not go deeper into why that was. Will definitely give it a try again in the future
  - randlet 2y ago
    Pyinstaller can be a bit fiddly to get right initially if you have a lot of package dependencies, but other than that it works really well.
    - Numerlor 2y ago
      Using the spec files for persistent readable configuration also goes a long way, if you treat pyinstaller as a python module you can automate it whole with just python, including the spec files as it executes them as python scripts I've had automated builds running mostly untouched for years here https://github.com/Numerlor/Auto_Neutron/blob/master/pyinstaller_build https://github.com/Numerlor/Auto_Neutron/blob/master/pyinsta..., though it doesn't have any particularly troublesome packages other than PySide6
  - WhyCause 2y ago
    PyInstaller-made executables also used to have a habit of getting flagged by security software as malicious (maybe that's why you couldn't run it?) -- apparently, so many malware writers used it that it ruined the party for everyone. Fortunately, that was only the 32-bit version of Python 2.7. Using 64-bit versions or Python 3 was enough to not get flagged as malicious. I figured that out when I decided I didn't want to teach myself Go just then to deploy something that had worked the day before.

BiteCode_dev 2y ago

I know it's not very nice to post that here, but given it's hard to get good info about python because the ecosystem is so huge, I'd rather give it straight: Don't use pex, use shiv: https://shiv.readthedocs.io/ https://shiv.readthedocs.io/ Pex doesn't work on windows, is slower than shiv and shiv solves the resource files problem that you will encounter with dependencies like django. To achieve this, shiv unzips the whole zipapp in a cache on disk and create a transparent proxy to it. Still, I wish zipapps had better tooling honestly. Using either pex or shiv is more complex than it needs to be. I hope uv will eventually give you something like "uv zipapp" and make it good and simple. Right now, to bundle all deps for shiv you have to download first all wheels, then figure out the proper incantation to include them. And if you use WSGI, you probably want some kind of dedicated entry point. The tech is cool, but it deserves maturing to be a true ".war for python".

fsloth 2y ago
”Pex doesn't work on windows” Oh that’s such a weird limitation. In the early 2000’s open source hubris, next year was always the year of the Linux desktop. Since then consumer windows matured into a totally ok OS and one with the best support for graphics and C++ development at that. Non-windows support nowadays is a fairly strong signal of a non-serious software offering if there is no obvious reason for it. And that’s totally fine, hobby tools developed by enthusiasts rock - but they are not industrial in scope as such.
- sneed_chucker 2y ago
  A lot of serious software offerings are only concerned with the server use case, modern servers run Linux unless there's a good reason not to, and modern windows has more than one acceptable way to run Linux binaries if you absolutely have to.
  - Vegenoid 2y ago
    > modern servers run Linux unless there's a good reason not to I think many people on HN would be surprised at how many orgs are using Windows servers heavily, because of their familiarity and comfort with Microsoft, or because some application requires it. Of course they are non-tech companies using the servers internally for enterprise applications, not web servers, but there is absolutely a lot of windows server usage in corporate environments.

serjester 2y ago

I think the easiest way to do this now is with uv and their support for inline metadata. At this point you can just have the user install uv with a single command and then have the user run the script with `uv run example.py`.

d0mine 2y ago
—script appears to be missing if you meant https://peps.python.org/pep-0723/ https://peps.python.org/pep-0723/

khaledh 2y ago

We've been using pex for a long while to package PySpark jobs with dependencies into a single file. Saves a ton of time vs. the old way of building/deploying docker images. https://spark.apache.org/docs/latest/api/python/user_guide/python_packaging.html#using-pex https://spark.apache.org/docs/latest/api/python/user_guide/p...

ttyprintk 2y ago
We shiv all the dependencies into a single file, bake it into a Docker layer, and refer to it in each scripts shebang.
fire_lake 2y ago
I would prefer Docker images since you they are easier to “cross compile”. However EMR requires the same Docker image for the entire cluster - why not per job!?!?
- khaledh 2y ago
  That's fair. Pex does support specifying a target platform though, which can package linux binary dependencies on a Mac for example. That being said, if a native dependency doesn't provide binaries (i.e. source only) we're out of luck. We solve this issue by precompiling and publishing those dependencies on a private package index. Thankfully there are only a handful of dependencies that need this.
  - fire_lake 2y ago
    Unfortunately it’s the most important dependencies that tend to be native - Spark, TensorFlow, Torch, etc…

gavmor 2y ago

I can see how .pex or shiv's zippaps live at the Pareto threshold where converting a Python app to an 80% (or less...) executable takes only 20% (or less!) of the effort that would be spent re-writing the app in eg C, C++, Rust, or Golang. Still, it breaks my heart to see all this work being done to keep developers employed as "Python developers", instead of leveling them up to a broader programming skillset. It seems as if the industry is investing rather heavily in a local maximum. But, of course, the market is has information that I don't.

Numerlor 2y ago
Eh I wouldn't call PEX proper executables that'd replace the languages above as they still need python, but I still prefer python for things like simple Qt apps even if it means using e.g. PyInstaller
slotrans 2y ago
It is often the case that the nifty Python thing you want to pass around uses one or more nifty Python libraries that have no C/C++/Rust/Golang equivalent (or no obvious equivalent), and so rewriting it becomes a herculean task.
- gavmor 2y ago
  I wonder if this isn't reflective of some universal law of niftiness.
zem 2y ago
> Still, it breaks my heart to see all this work being done to keep developers employed as "Python developers", instead of leveling them up to a broader programming skillset. It seems as if the industry is investing rather heavily in a local maximum. this is a topic near to my heart; I have spent the last several years building my career around doing this sort of work (specifically writing tools to expand the capabilities of python code). the key idea is not that we "keep people employed as python developers", it is that python (you can also substitute ruby, javascript, php, or any other popular dynamic language) is a genuinely powerful and expressive language from the point of view of converting ideas to code, and having that code be readable, maintainable, and extensible by other humans. where is falls down is on the "machine facing" end - you don't get the guarantees of static typing, the import system is a bit of a mess, it isn't as fast as lower level, statically-compiled-to-native-code languages, and (as the current discussion shows) packaging and distributing an end user binary is somewhere between a morass and a nightmare. however, none of these things are a problem with the language per se, and they can be fixed with some combination of improving the implementation, writing better tooling (cf valgrind from the c world), making it easier to write small bits of critical code in c/rust/zig/etc, and improving language features while not sacrificing what draws people to use python in the first place. is python a local maximum? almost certainly; it would be foolish to think that nothing better could come along, and indeed pyret already looks distinctly better in terms of pure dynamic language design. but the answer to that is not to "level up" people who are happy and productive using python to some language like rust or c++ that imposes a heavier cognitive burden on the user to take care of low level details that their applications might not need, or one like golang that pushes the complexity out into the application where problems need to be solved via design patterns that people need to learn and copy, rather than just building them into the language or stdlib.

mappu 2y ago

Pex and Shiv both require the end user to have Python installed, so I think PyOxidizer is the best tool in this category: https://pyoxidizer.readthedocs.io/en/stable/pyoxidizer_comparisons.html https://pyoxidizer.readthedocs.io/en/stable/pyoxidizer_compa...

martsa1 2y ago
How does Pyoxidizer compare to pyinstaller and nuitka?
- mappu 2y ago
  Great question! That was actually covered in the link I posted. Compared to PyInstaller, it doesn't dump any temporary files on disk. Compared to Nuitka, it uses the real CPython, not a different implementation.
stuhood 2y ago
Not anymore: PEX now supports including an interpreter in the package: https://docs.pex-tool.org/scie.html https://docs.pex-tool.org/scie.html And unfortunately, PyOxidizer went into maintenance mode: https://gregoryszorc.com/blog/2024/03/17/my-shifting-open-source-priorities/ https://gregoryszorc.com/blog/2024/03/17/my-shifting-open-so...
delijati 2y ago
Adding the interpreter is fairly easy. I had in project to do it for windows. If someone is interested. https://github.com/delijati/docker-shiv?tab=readme-ov-file#add-python-interpreter https://github.com/delijati/docker-shiv?tab=readme-ov-file#a...

smallmouth 2y ago

I thought this was going to be a post about plumbing and then I read further and was even more excited to say the phrase "python executable". Very interesting. Thanks!

jbritton 2y ago

I use cx_Freeze to get an exe and Inno Setup for a Windows installer. It has worked well for me. I also have a DigiCert dongle to sign the exes.

NeutralForest 2y ago

I've quickly skimmed the comments and haven't found a mention of briefcase (https://beeware.org/project/projects/tools/briefcase/ https://beeware.org/project/projects/tools/briefcase/) which is a tool to build standalone apps in Python.

bborud 2y ago

The comments remind me of the Judean People’s Front scene from Life of Brian. Everyone has their solution to the problem - so there is no solution.

xrd 2y ago

I just put everything inside a docker container. But those 9GB containers when you use pytorch and Nvidia cuda libraries. Has anyone tried using this for that type of packaging?

shalabhc 2y ago

We have been using pex to deploy code to running docker containers in ECS to avoid the cold start delay. Cuts down the iteration loop time for development significantly. https://dagster.io/blog/fast-deploys-with-pex-and-docker https://dagster.io/blog/fast-deploys-with-pex-and-docker