4 ms·

Virtual environments are cool, and necessary, but at the same time, they are incredibly limited and I always get frustrated at the lack of features they should

Virtual environments are cool, and necessary, but at the same time, they are incredibly limited and I always get frustrated at the lack of features they should have. They are too fragile. Nuking a whole venv when you mess up isn't really efficient. They aren't portable. You need to package up your editable project for an offline system? Too bad, virtual environments use hardcoded paths and symlinks that will be broken when you try. Want to convert the packages in your venv back to .tar.gz or wheels? There's no way to do that either.

doctorpangloss 2y ago
> There's no concept of caching venv packages elsewhere A venv's pip shares a systemwide cache for downloads.
- sealedservant 2y ago
  Ah, that's news to me! Updated previous comment for accuracy. I mostly use pipx these days for stuff I need on the PATH.
slt2021 2y ago
Dockerfile and requirements.txt is everything you need to know to recreate environment, and this is how apps are usually ran in production. This is how you can create consistent recreatable and testable env and carry it from dev machine to test, staging, CI/CD, and prod environments Never really had an issue with python apps not being portable. Building regular python app? Just use official python docker image. Using advanced CUDA stuff with nvidia cards? Just use nvidia’s docker image and forget about fiddling with configuring and compiling dependencies. It is all as easy as carrying dockerfile and requirements.txt
- OutOfHere 2y ago
  I dockerize, but using devcontainer.json, and this also plays well with VScode.
Scaevolus 2y ago
Pip will cache the downloaded package tarballs to make repeated installs faster at least, but that's little help when you have a dozen venvs with multiple gigabytes of pytorch.
- hadlock 2y ago
  Time to containerize your pytorch app
rangerelf 2y ago
I've read "they're too/so/very fragile" so many times, but nobody has given me an example of how or why they consider them fragile, what steps do I need to do to break them. Myself? I consider them ephemeral: I create my Makefile with a 'venv' target to delete/rebuild the virtual environment in case of changes. Some do take longer to rebuild, but with a global pip cache it takes much less time to rebuild once it's been done the first time. Hardcoded paths and symlinks? Not a problem, I know that virtual environments don't travel; and since they're ephemeral if I need it at a different location I can always rebuild them. Do you have an example for your third paragraph? And I've done the last one, converted all packages to wheels, uploaded to an artifact server, and used those for production deployments. I'm curious, really; not bashing you for having troubles with it, but I don't understand the aversion given my lack of blockers when using them.
- slt2021 2y ago
  Can I ask you a question, what do you think is better approach: 1) publish packages as wheels or 2) publish apps as docker images (or docker-compose files or helm charts)? Why some People prefer 1 to 2? I think 2 is more “production friendly” and universal across other languages and stacks (same approach used for java js ruby etc)
  - lexicality 2y ago
    How would you use the python `requests` library if it was packaged as a docker image?
    - slt2021 2y ago
      I meant you package your whole app into container with all required libs, and dont publish your app as a package
  - rangerelf 2y ago
    Vastly different use cases. Publish a module as a wheel if you want to be able to use pip to install it in your servers, or anywhere really. Once you've built out your virtual environment you can build your docker image and push it to a registry. Your helm chart should include references to docker images that are getting deployed to a kubernetes cluster, among other entities that need to be built out for your app to work. Those are all different layers of the deployment, independent of each other. If someone prefers 1 over 2 (or viceversa), they don't understand what they think they know.
spapas82 2y ago
> They aren't portable. You need to package up your editable project for an offline system? Too bad, virtual environments use hardcoded paths and symlinks that will be broken when you try. This isn't 100% true. If you carefully use the same python version and the same path for your venv and python then copying the venv between computers works perfectly. I've done it many times, actually last time I changed my computer I copied over all my projects and venvs and everything worked flawlessly.

doctorpangloss 2y ago

> There's no concept of caching venv packages elsewhere A venv's pip shares a systemwide cache for downloads.

sealedservant 2y ago
Ah, that's news to me! Updated previous comment for accuracy. I mostly use pipx these days for stuff I need on the PATH.

slt2021 2y ago

Dockerfile and requirements.txt is everything you need to know to recreate environment, and this is how apps are usually ran in production. This is how you can create consistent recreatable and testable env and carry it from dev machine to test, staging, CI/CD, and prod environments Never really had an issue with python apps not being portable. Building regular python app? Just use official python docker image. Using advanced CUDA stuff with nvidia cards? Just use nvidia’s docker image and forget about fiddling with configuring and compiling dependencies. It is all as easy as carrying dockerfile and requirements.txt

OutOfHere 2y ago
I dockerize, but using devcontainer.json, and this also plays well with VScode.

Scaevolus 2y ago

Pip will cache the downloaded package tarballs to make repeated installs faster at least, but that's little help when you have a dozen venvs with multiple gigabytes of pytorch.

hadlock 2y ago
Time to containerize your pytorch app

rangerelf 2y ago

I've read "they're too/so/very fragile" so many times, but nobody has given me an example of how or why they consider them fragile, what steps do I need to do to break them. Myself? I consider them ephemeral: I create my Makefile with a 'venv' target to delete/rebuild the virtual environment in case of changes. Some do take longer to rebuild, but with a global pip cache it takes much less time to rebuild once it's been done the first time. Hardcoded paths and symlinks? Not a problem, I know that virtual environments don't travel; and since they're ephemeral if I need it at a different location I can always rebuild them. Do you have an example for your third paragraph? And I've done the last one, converted all packages to wheels, uploaded to an artifact server, and used those for production deployments. I'm curious, really; not bashing you for having troubles with it, but I don't understand the aversion given my lack of blockers when using them.

slt2021 2y ago
Can I ask you a question, what do you think is better approach: 1) publish packages as wheels or 2) publish apps as docker images (or docker-compose files or helm charts)? Why some People prefer 1 to 2? I think 2 is more “production friendly” and universal across other languages and stacks (same approach used for java js ruby etc)
- lexicality 2y ago
  How would you use the python `requests` library if it was packaged as a docker image?
  - slt2021 2y ago
    I meant you package your whole app into container with all required libs, and dont publish your app as a package
- rangerelf 2y ago
  Vastly different use cases. Publish a module as a wheel if you want to be able to use pip to install it in your servers, or anywhere really. Once you've built out your virtual environment you can build your docker image and push it to a registry. Your helm chart should include references to docker images that are getting deployed to a kubernetes cluster, among other entities that need to be built out for your app to work. Those are all different layers of the deployment, independent of each other. If someone prefers 1 over 2 (or viceversa), they don't understand what they think they know.

spapas82 2y ago

> They aren't portable. You need to package up your editable project for an offline system? Too bad, virtual environments use hardcoded paths and symlinks that will be broken when you try. This isn't 100% true. If you carefully use the same python version and the same path for your venv and python then copying the venv between computers works perfectly. I've done it many times, actually last time I changed my computer I copied over all my projects and venvs and everything worked flawlessly.