4 ms·

The fact that the suggested solution in Python is to give every Python script a full copy of an entire specific Python runtime (via venv) is a mild annoyance as

The fact that the suggested solution in Python is to give every Python script a full copy of an entire specific Python runtime (via venv) is a mild annoyance as a design pattern...to me. Python scripting today requires shipping your development environment. Python is wonderful until you want to run that code on another machine. At that point, the target system has to venv their way into reproducing your environment...often including the specific Python interpreter you picked, and to download (and possibly compile) all modules and their dependencies. It can work beautifully and many of the large companies I've worked for have put oodles of engineering effort into making it "easy", as long as you follow their happy path and don't deviate. But, to your point, on my own machine with venv + pip it "just works". The pain comes when trying to venv + pip on another machine. How confident are you that you could venv + pip your moderately complex Python application on 200k machines without issue? What if it's a mix of Windows, Linux, and macOS? (this is a real scenario I've experienced). From my own experience I can share that it is painful. Your mileage may vary. And after a while your system is riddled with Python venvs. Leading to some mildly annoying problems with popular IDE's: https://www.jetbrains.com/help/pycharm/package-installation-issues.html https://www.jetbrains.com/help/pycharm/package-installation-...

elbigbad 4y ago
Yeah, I agree with this. I typically dockerize where appropriate to sidestep this a little, but very often rewrite everything into a more friendly language like Go if I am intending to ship software to other machines. The latter is obviously painful if I’m making heavy use of Python specific libraries that do a lot of heavy lifting like Numpy, if those features aren’t in something like GoNum.
- behnamoh 4y ago
  I should also say: Unfortunately, Conda’s environment exports only work for similar OS’s. I don’t know why Python’s situation is like this, after so many years. Does docker fix all the above issues?
  - deleted 4y ago
    [deleted]
  - bornfreddy 4y ago
    Yes, docker solves all these issues (because your app carries all of its environment in the image), however it is a bit cumbersome if you want to distribute your app to many unknown users. They can't just download something and run it, as Windows users are used to doing.
  - bloblaw 4y ago
    Docker simultaneously solves and creates problems here. Step 1: Docker must be installed on all targets. This is not a given and is a new piece of overhead. Step 2: The Dockerfile hopefully doesn't source from just "ubuntu:latest" and bring in the whole kitchen sink for this SINGLE APPLICATION. Effectively if you are using Docker and deploying your script to Windows or macOS, you are saying "Hey, this script requires you to install linux in a VM to run this. Docker makes it easy. Go download 500mb of Docker and a couple hundred more mb of images to run this 800k script.
andreareina 4y ago
Ansible is a special example of this. Some Ansible modules (e.g. the postgresql one) require non-stdlib modules to be installed in the remote python. So either install it globally or figure out how to make Ansible work with environments. Even if the thing you're trying to set up ostensibly doesn't use python. sigh
- danudey 4y ago
  As I recall, pipx does this; it installs a separate env, but then provides entrypoints (commands) to the global environment. For example, it would install ansible and the ansible dependencies to a separate environment somewhere, but still put commands for `ansible`, `ansible-playbook`, etc. into /usr/local/bin for example. When you run the ansible command, it loads ansible from the separate environment. I haven't actually tried this myself since our environments are pretty uniform, but it could be worth looking into. It would be nice if pip itself provided this as an option (or if it became the default), but that could also become extremely complicated in terms of upgrades. I already hate having to download and build scipy, but it would be incredibly irritating to have to do it once for every tool I have that uses it. I'm sure some kind of cache could be made to work, but it's not trivial.
  - andreareina 4y ago
    Big fan of pipx, but I don't think it would work here? Local ansible calls remote python directly and that's where the modules need to be, so a sandboxed {local,remote} ansible would be inert. There's a workaround but it's not ergonomic and I always forget what it is.
na85 4y ago
>The fact that the suggested solution in Python is to give every Python script a full copy of an entire specific Python runtime (via venv) is a mild annoyance as a design pattern...to me. It's not just a mild annoyance, it's a sad statement about python as a community of developers that this is not only accepted but recommended. I write python because some of what I do occurs in the domains where python makes the most sense, but the idea that the python community accepts venv rather than considering the fact that venv even exists at all to be a source of profound embarassment is mystifying to me.
thaumasiotes 4y ago
> The fact that the suggested solution in Python is to give every Python script a full copy of an entire specific Python runtime (via venv) is a mild annoyance as a design pattern...to me. That bothers me too, a bit. But sidethread to you people are comparing python unfavorably to ruby, which as far as I know uses exactly the same strategy (except that it's called "rvm" instead of "virtualenv"). Node.js appears to do the same thing. To my eyes, the solution to this would be static linking, but I don't know how much sense that makes in the python / ruby / js context. Other than that, what's the alternative? There's certainly a lot of convergence on this solution.
mixmastamyk 4y ago
Not necessary unless project is huge. And not a full copy, venv uses links. Pip will install copies if you ask.
okasaki 4y ago
Virtualenvs are no worse than static linking, or packaging dlls/sos, all very common practice (and even recommended). Python package management is VERY easy.
- bloblaw 4y ago
  When I statically compile a C++ or Go binary, I end up with something I can copy and execute on any platform for which it was compiled for. Single file. Copy and run. Virtualenvs require every single target to reproduce your dev environment: (1) have internet access and able to reach pypi (or artifactory, or whatever you use). (2) the ability to install the required version of Python if it isn't already installed. That's another 30mb download. Ever want to run a complex Python script on a bastion host or a host behind a bastion? Well now #1 and #2 above won't work (or haven't in my direct experience working for some of the big cloud companies). So you have to use something like PyInstaller and hope it works. It might. It might not. A statically compiled C++ binary or Go binary probably will. This thread is full of pain points in python package management. The first step is admitting there is a problem. I don't think you've experienced the pain caused by Python package managers that others outline in this thread.
  - okasaki 4y ago
    I mean in the sense that you're bundling your dependencies with your program. This is well established, Python isn't special. Obviously Python is (usually) an interpreted language, so you're going to need Python on user system. If that's a problem Python might not be for you, or you're going to need to do some extra work. It can be done. For example the popular visual novel software Ren'Py is written in Python.
SAI_Peregrinus 4y ago
`pyinstaller --onefile <script.py> [secondscript.py] [...]` Statically links all the dependencies and the Python interpreter. Other machine can just run the executable, no need to set up a venv+pip. Of course the binary is huge, but IME it works. Unfortunately it can't cross-build, so you need a VM or multiple machines set up with the dev environment to run pyinstaller for each OS.

elbigbad 4y ago

Yeah, I agree with this. I typically dockerize where appropriate to sidestep this a little, but very often rewrite everything into a more friendly language like Go if I am intending to ship software to other machines. The latter is obviously painful if I’m making heavy use of Python specific libraries that do a lot of heavy lifting like Numpy, if those features aren’t in something like GoNum.

behnamoh 4y ago
I should also say: Unfortunately, Conda’s environment exports only work for similar OS’s. I don’t know why Python’s situation is like this, after so many years. Does docker fix all the above issues?
- deleted 4y ago
  [deleted]
- bornfreddy 4y ago
  Yes, docker solves all these issues (because your app carries all of its environment in the image), however it is a bit cumbersome if you want to distribute your app to many unknown users. They can't just download something and run it, as Windows users are used to doing.
- bloblaw 4y ago
  Docker simultaneously solves and creates problems here. Step 1: Docker must be installed on all targets. This is not a given and is a new piece of overhead. Step 2: The Dockerfile hopefully doesn't source from just "ubuntu:latest" and bring in the whole kitchen sink for this SINGLE APPLICATION. Effectively if you are using Docker and deploying your script to Windows or macOS, you are saying "Hey, this script requires you to install linux in a VM to run this. Docker makes it easy. Go download 500mb of Docker and a couple hundred more mb of images to run this 800k script.

andreareina 4y ago

Ansible is a special example of this. Some Ansible modules (e.g. the postgresql one) require non-stdlib modules to be installed in the remote python. So either install it globally or figure out how to make Ansible work with environments. Even if the thing you're trying to set up ostensibly doesn't use python. sigh

danudey 4y ago
As I recall, pipx does this; it installs a separate env, but then provides entrypoints (commands) to the global environment. For example, it would install ansible and the ansible dependencies to a separate environment somewhere, but still put commands for `ansible`, `ansible-playbook`, etc. into /usr/local/bin for example. When you run the ansible command, it loads ansible from the separate environment. I haven't actually tried this myself since our environments are pretty uniform, but it could be worth looking into. It would be nice if pip itself provided this as an option (or if it became the default), but that could also become extremely complicated in terms of upgrades. I already hate having to download and build scipy, but it would be incredibly irritating to have to do it once for every tool I have that uses it. I'm sure some kind of cache could be made to work, but it's not trivial.
- andreareina 4y ago
  Big fan of pipx, but I don't think it would work here? Local ansible calls remote python directly and that's where the modules need to be, so a sandboxed {local,remote} ansible would be inert. There's a workaround but it's not ergonomic and I always forget what it is.

na85 4y ago

>The fact that the suggested solution in Python is to give every Python script a full copy of an entire specific Python runtime (via venv) is a mild annoyance as a design pattern...to me. It's not just a mild annoyance, it's a sad statement about python as a community of developers that this is not only accepted but recommended. I write python because some of what I do occurs in the domains where python makes the most sense, but the idea that the python community accepts venv rather than considering the fact that venv even exists at all to be a source of profound embarassment is mystifying to me.

thaumasiotes 4y ago

> The fact that the suggested solution in Python is to give every Python script a full copy of an entire specific Python runtime (via venv) is a mild annoyance as a design pattern...to me. That bothers me too, a bit. But sidethread to you people are comparing python unfavorably to ruby, which as far as I know uses exactly the same strategy (except that it's called "rvm" instead of "virtualenv"). Node.js appears to do the same thing. To my eyes, the solution to this would be static linking, but I don't know how much sense that makes in the python / ruby / js context. Other than that, what's the alternative? There's certainly a lot of convergence on this solution.

mixmastamyk 4y ago

Not necessary unless project is huge. And not a full copy, venv uses links. Pip will install copies if you ask.

okasaki 4y ago

Virtualenvs are no worse than static linking, or packaging dlls/sos, all very common practice (and even recommended). Python package management is VERY easy.

bloblaw 4y ago
When I statically compile a C++ or Go binary, I end up with something I can copy and execute on any platform for which it was compiled for. Single file. Copy and run. Virtualenvs require every single target to reproduce your dev environment: (1) have internet access and able to reach pypi (or artifactory, or whatever you use). (2) the ability to install the required version of Python if it isn't already installed. That's another 30mb download. Ever want to run a complex Python script on a bastion host or a host behind a bastion? Well now #1 and #2 above won't work (or haven't in my direct experience working for some of the big cloud companies). So you have to use something like PyInstaller and hope it works. It might. It might not. A statically compiled C++ binary or Go binary probably will. This thread is full of pain points in python package management. The first step is admitting there is a problem. I don't think you've experienced the pain caused by Python package managers that others outline in this thread.
- okasaki 4y ago
  I mean in the sense that you're bundling your dependencies with your program. This is well established, Python isn't special. Obviously Python is (usually) an interpreted language, so you're going to need Python on user system. If that's a problem Python might not be for you, or you're going to need to do some extra work. It can be done. For example the popular visual novel software Ren'Py is written in Python.

SAI_Peregrinus 4y ago

`pyinstaller --onefile <script.py> [secondscript.py] [...]` Statically links all the dependencies and the Python interpreter. Other machine can just run the executable, no need to set up a venv+pip. Of course the binary is huge, but IME it works. Unfortunately it can't cross-build, so you need a VM or multiple machines set up with the dev environment to run pyinstaller for each OS.