5 ms·

I know it's not very nice to post that here, but given it's hard to get good info about python because the ecosystem is so huge, I'd rather give it straight: D

I know it's not very nice to post that here, but given it's hard to get good info about python because the ecosystem is so huge, I'd rather give it straight: Don't use pex, use shiv: https://shiv.readthedocs.io/ https://shiv.readthedocs.io/ Pex doesn't work on windows, is slower than shiv and shiv solves the resource files problem that you will encounter with dependencies like django. To achieve this, shiv unzips the whole zipapp in a cache on disk and create a transparent proxy to it. Still, I wish zipapps had better tooling honestly. Using either pex or shiv is more complex than it needs to be. I hope uv will eventually give you something like "uv zipapp" and make it good and simple. Right now, to bundle all deps for shiv you have to download first all wheels, then figure out the proper incantation to include them. And if you use WSGI, you probably want some kind of dedicated entry point. The tech is cool, but it deserves maturing to be a true ".war for python".

fsloth 2y ago
”Pex doesn't work on windows” Oh that’s such a weird limitation. In the early 2000’s open source hubris, next year was always the year of the Linux desktop. Since then consumer windows matured into a totally ok OS and one with the best support for graphics and C++ development at that. Non-windows support nowadays is a fairly strong signal of a non-serious software offering if there is no obvious reason for it. And that’s totally fine, hobby tools developed by enthusiasts rock - but they are not industrial in scope as such.
- sneed_chucker 2y ago
  A lot of serious software offerings are only concerned with the server use case, modern servers run Linux unless there's a good reason not to, and modern windows has more than one acceptable way to run Linux binaries if you absolutely have to.
  - Vegenoid 2y ago
    > modern servers run Linux unless there's a good reason not to I think many people on HN would be surprised at how many orgs are using Windows servers heavily, because of their familiarity and comfort with Microsoft, or because some application requires it. Of course they are non-tech companies using the servers internally for enterprise applications, not web servers, but there is absolutely a lot of windows server usage in corporate environments.
    - wolrah 2y ago
      > I think many people on HN would be surprised at how many orgs are using Windows servers heavily, because of their familiarity and comfort with Microsoft, or because some application requires it. I think most HN readers are well aware that there are a lot of Windows servers out there, especially in the sorts of environments where it's "The Server". That doesn't change the fact that there are orders of magnitude more Linux servers in the world, and as the post you replied to said Linux is the default assumption. Basically every container and the vast majority of VM guests are Linux. I'd be willing to bet that more Linux servers have been deployed in the time it took me to type this post than Windows servers will be deployed this week.
      - Vegenoid 2y ago
        I'm not trying to make the point that there is a comparable number of Windows servers to Linux servers, and I believe that going off "number of servers" is of limited usefulness. There are of course going to be far more Linux servers because they are far easier to provision, and the people managing them are generally going to be much more apt at orchestrating large fleets of small servers than the people managing Windows servers, which will tend towards larger servers with multiple purposes. I'm simply trying to say that Linux is not the default assumption in many contexts where a lot of money gets spent on licenses and services for server software. There are of course large contexts (tech companies and web servers) where Linux is the default assumption.
- murph 2y ago
  PEX was originally developed by Twitter, and was used heavily within the company. Twitter ran primarily OSX (for devs) and Linux (for servers).
- two_handfuls 2y ago
  It's not that weird, because Windows is the most different of the common OS so it's the hardest to make work. I agree though that Windows support is an important feature for a tool like this, and it's not complete without it.
MikeTheGreat 2y ago
While we're posting not-nice things, I'll throw my current favorite out: pipx ( https://github.com/pypa/pipx https://github.com/pypa/pipx ). I use Python's Poetry to build a wheel and then pipx to install it. Super-easy, barely an inconvenience :)
- BiteCode_dev 2y ago
  pipx is orthogonal to the problem pex and shiv are solving since: - it doesn't provide a single file for a full project with all dependencies. - it requires internet access on the target machine to install things. - it can't give you access to underlying API for things like wsgi. - it requires pipx on the target machine. - it requires a wheels made from a build, which many projects don't have (such as a lot of web projects).
  - MikeTheGreat 2y ago
    Hello! I'm not a pipx expert (I'm a "I got it to work for my use case and then stopped" person :) ) but I was able to get my project working after installing a local wheel file (so i don't think you need access to the Internet to install things). I thought the wheel file has all the dependencies since (my understanding of) pipx uses a venv for each individual wheel file/app. Certainly I'm using a bunch of packages that I needed to pip install into my development environment, but it's entirely possible that my pipx-managed wheels are using those installs, too. I think the last two points are accurate, and I've never tried to use APIs like wsgi with it.
  - lordmauve 2y ago
    Also pipx isn't reproducible - it re-resolves dependencies so you may end up with different versions over time or in different places, eventually causing something to break. If you have a shiv working it stays working, assuming you have a solution to distribute the required interpreter version.
packetlost 2y ago
I feel like, in general, you should be able to treat an archive as a filesystem. Zipapps could be so much easier if it did that transparently.
- BiteCode_dev 2y ago
  It can, but: - It's slow. - The underlying python code cannot use open() to access its own resources, but must use pkg_resources, and most packages on pypi don't do that because most dev don't even know it exists or what problem is soves. - _ _ file _ _ is not available, breaking a lot of things. zip_safe is a build option, but actual compliance to it is hard work and rare.
  - packetlost 2y ago
    I was thinking more along the lines of using namespaces and FUSE. I'm aware that Python kinda treats them like folders transparently, but not completely for reasons (and more) that you described.
    - BiteCode_dev 2y ago
      That's a cool idea. Wonder what the FUSE equivalent for windows is.
      - packetlost 2y ago
        Probably Windows Projected FileSystem: https://learn.microsoft.com/en-us/windows/win32/projfs/projected-file-system https://learn.microsoft.com/en-us/windows/win32/projfs/proje...
      - networked 2y ago
        There seem to be two popular FUSE-like things for Windows: https://github.com/dokan-dev/dokany https://github.com/dokan-dev/dokany and https://github.com/winfsp/winfsp https://github.com/winfsp/winfsp. Both implement a FUSE compatibility API in addition to their own API. A 2016 comparison by the developer of WinFsp says WinFsp is several times faster in performance tests [1]. According to the readme for Donaky [2], since then Dokany got faster. [1] https://github.com/winfsp/winfsp/issues/19#issuecomment-265368159 https://github.com/winfsp/winfsp/issues/19#issuecomment-2653... [2] https://github.com/dokan-dev/dokany/blob/255e4a508b744996191d652af70d912c26c74f56/README.md#benchmark-v1511000-vs-v2031000 https://github.com/dokan-dev/dokany/blob/255e4a508b744996191...
dbaupp 2y ago
FWIW, pex now also has options to unzip the archive to a cache directory on startup (I believe this happens by default now, but am not at a computer to confirm), to side step the zipapp limitations that you reference.
- BiteCode_dev 2y ago
  I just checked, and there is indeed a `--pex-root` option with and even "-c" to specify a custom entry point. Thanks for pointing it out.
emmelaich 2y ago
> .war for python It shouldn't be that hard to get a general 'jar/war', not just for python. Perhaps jart's cosmolitan libc with a zip file appendage which it auto extracts and optionally caches on first use. A heavyweight solution, but still lighter than Docker.
- BiteCode_dev 2y ago
  It would today, because macos requires signing, and windows has "the mark of the web". They don't apply to scripts, so Python is fine, but they do apply to executables. Having to sign your zip every time you produce it would nullify the benefit of the concept. And because the checksum of the executable file change according to the content of the zip, you can't just sign the cosmopolitan part.
  - emmelaich 2y ago
    You can work around both, but I agree that's a big point of friction.
ttyprintk 2y ago
Pex works with older versions than shiv.
delijati 2y ago
Adding the interpreter is fairly easy. I had in project to do it for windows. If someone is interested. https://github.com/delijati/docker-shiv?tab=readme-ov-file#add-python-interpreter https://github.com/delijati/docker-shiv?tab=readme-ov-file#a...

fsloth 2y ago

”Pex doesn't work on windows” Oh that’s such a weird limitation. In the early 2000’s open source hubris, next year was always the year of the Linux desktop. Since then consumer windows matured into a totally ok OS and one with the best support for graphics and C++ development at that. Non-windows support nowadays is a fairly strong signal of a non-serious software offering if there is no obvious reason for it. And that’s totally fine, hobby tools developed by enthusiasts rock - but they are not industrial in scope as such.

sneed_chucker 2y ago
A lot of serious software offerings are only concerned with the server use case, modern servers run Linux unless there's a good reason not to, and modern windows has more than one acceptable way to run Linux binaries if you absolutely have to.
- Vegenoid 2y ago
  > modern servers run Linux unless there's a good reason not to I think many people on HN would be surprised at how many orgs are using Windows servers heavily, because of their familiarity and comfort with Microsoft, or because some application requires it. Of course they are non-tech companies using the servers internally for enterprise applications, not web servers, but there is absolutely a lot of windows server usage in corporate environments.
  - wolrah 2y ago
    > I think many people on HN would be surprised at how many orgs are using Windows servers heavily, because of their familiarity and comfort with Microsoft, or because some application requires it. I think most HN readers are well aware that there are a lot of Windows servers out there, especially in the sorts of environments where it's "The Server". That doesn't change the fact that there are orders of magnitude more Linux servers in the world, and as the post you replied to said Linux is the default assumption. Basically every container and the vast majority of VM guests are Linux. I'd be willing to bet that more Linux servers have been deployed in the time it took me to type this post than Windows servers will be deployed this week.
    - Vegenoid 2y ago
      I'm not trying to make the point that there is a comparable number of Windows servers to Linux servers, and I believe that going off "number of servers" is of limited usefulness. There are of course going to be far more Linux servers because they are far easier to provision, and the people managing them are generally going to be much more apt at orchestrating large fleets of small servers than the people managing Windows servers, which will tend towards larger servers with multiple purposes. I'm simply trying to say that Linux is not the default assumption in many contexts where a lot of money gets spent on licenses and services for server software. There are of course large contexts (tech companies and web servers) where Linux is the default assumption.
murph 2y ago
PEX was originally developed by Twitter, and was used heavily within the company. Twitter ran primarily OSX (for devs) and Linux (for servers).
two_handfuls 2y ago
It's not that weird, because Windows is the most different of the common OS so it's the hardest to make work. I agree though that Windows support is an important feature for a tool like this, and it's not complete without it.

MikeTheGreat 2y ago

While we're posting not-nice things, I'll throw my current favorite out: pipx ( https://github.com/pypa/pipx https://github.com/pypa/pipx ). I use Python's Poetry to build a wheel and then pipx to install it. Super-easy, barely an inconvenience :)

BiteCode_dev 2y ago
pipx is orthogonal to the problem pex and shiv are solving since: - it doesn't provide a single file for a full project with all dependencies. - it requires internet access on the target machine to install things. - it can't give you access to underlying API for things like wsgi. - it requires pipx on the target machine. - it requires a wheels made from a build, which many projects don't have (such as a lot of web projects).
- MikeTheGreat 2y ago
  Hello! I'm not a pipx expert (I'm a "I got it to work for my use case and then stopped" person :) ) but I was able to get my project working after installing a local wheel file (so i don't think you need access to the Internet to install things). I thought the wheel file has all the dependencies since (my understanding of) pipx uses a venv for each individual wheel file/app. Certainly I'm using a bunch of packages that I needed to pip install into my development environment, but it's entirely possible that my pipx-managed wheels are using those installs, too. I think the last two points are accurate, and I've never tried to use APIs like wsgi with it.
- lordmauve 2y ago
  Also pipx isn't reproducible - it re-resolves dependencies so you may end up with different versions over time or in different places, eventually causing something to break. If you have a shiv working it stays working, assuming you have a solution to distribute the required interpreter version.

packetlost 2y ago

I feel like, in general, you should be able to treat an archive as a filesystem. Zipapps could be so much easier if it did that transparently.

BiteCode_dev 2y ago
It can, but: - It's slow. - The underlying python code cannot use open() to access its own resources, but must use pkg_resources, and most packages on pypi don't do that because most dev don't even know it exists or what problem is soves. - _ _ file _ _ is not available, breaking a lot of things. zip_safe is a build option, but actual compliance to it is hard work and rare.
- packetlost 2y ago
  I was thinking more along the lines of using namespaces and FUSE. I'm aware that Python kinda treats them like folders transparently, but not completely for reasons (and more) that you described.
  - BiteCode_dev 2y ago
    That's a cool idea. Wonder what the FUSE equivalent for windows is.
    - packetlost 2y ago
      Probably Windows Projected FileSystem: https://learn.microsoft.com/en-us/windows/win32/projfs/projected-file-system https://learn.microsoft.com/en-us/windows/win32/projfs/proje...
    - networked 2y ago
      There seem to be two popular FUSE-like things for Windows: https://github.com/dokan-dev/dokany https://github.com/dokan-dev/dokany and https://github.com/winfsp/winfsp https://github.com/winfsp/winfsp. Both implement a FUSE compatibility API in addition to their own API. A 2016 comparison by the developer of WinFsp says WinFsp is several times faster in performance tests [1]. According to the readme for Donaky [2], since then Dokany got faster. [1] https://github.com/winfsp/winfsp/issues/19#issuecomment-265368159 https://github.com/winfsp/winfsp/issues/19#issuecomment-2653... [2] https://github.com/dokan-dev/dokany/blob/255e4a508b744996191d652af70d912c26c74f56/README.md#benchmark-v1511000-vs-v2031000 https://github.com/dokan-dev/dokany/blob/255e4a508b744996191...

dbaupp 2y ago

FWIW, pex now also has options to unzip the archive to a cache directory on startup (I believe this happens by default now, but am not at a computer to confirm), to side step the zipapp limitations that you reference.

BiteCode_dev 2y ago
I just checked, and there is indeed a `--pex-root` option with and even "-c" to specify a custom entry point. Thanks for pointing it out.

emmelaich 2y ago

> .war for python It shouldn't be that hard to get a general 'jar/war', not just for python. Perhaps jart's cosmolitan libc with a zip file appendage which it auto extracts and optionally caches on first use. A heavyweight solution, but still lighter than Docker.

BiteCode_dev 2y ago
It would today, because macos requires signing, and windows has "the mark of the web". They don't apply to scripts, so Python is fine, but they do apply to executables. Having to sign your zip every time you produce it would nullify the benefit of the concept. And because the checksum of the executable file change according to the content of the zip, you can't just sign the cosmopolitan part.
- emmelaich 2y ago
  You can work around both, but I agree that's a big point of friction.

ttyprintk 2y ago

Pex works with older versions than shiv.

delijati 2y ago

Adding the interpreter is fairly easy. I had in project to do it for windows. If someone is interested. https://github.com/delijati/docker-shiv?tab=readme-ov-file#add-python-interpreter https://github.com/delijati/docker-shiv?tab=readme-ov-file#a...