6 ms·
Venvstacks: Virtual Environment Stacks for Python
- wizrrd 2y agoHow do venvstacks differ from PyInstaller and other bundlers?
- incognito124 2y agoSo, Docker container image, without OverlayFS benefits?
- zeotroph 2y agoAnd with just 3 layers: Runtime, Framework, Application. But at least you are not switching tools, and it presumably would prevent you from installing LARGE_v1.1, and then installing TINY_v2.2 in a later layer, which however upgrades LARGE to v1.2 and your docker images are now twice the size.
- davedx 2y agoPython’s Cambrian explosion of dependency management continues. Maybe one day via some strange machine form of genetic algorithms a canonical system will emerge.
- throwaway984393 2y ago[dead]
- salomonk_mur 2y agouv is clearly becoming the norm
- xelamonster 2y agoI was expecting this to be containers under the hood, instead it seems the idea is basically to bundle your virtualenv and ship it to prod? I think I'll stick with Docker...
- thangngoc89 2y agoI’m using poetry because of the default lock file usage (there isn’t a global, mutable state, all changes required to update the lock file). I wish that this function could be incorporated to poetry directly
- Terretta 2y agoFor a variety of reasons this season is a good time to look at `uv`. See this by the author of Rye: https://lucumr.pocoo.org/2024/8/21/harvest-season/ https://lucumr.pocoo.org/2024/8/21/harvest-season/ "Unified" packaging: https://astral.sh/blog/uv-unified-python-packaging https://astral.sh/blog/uv-unified-python-packaging
- trallnag 2y agoPoetry works so well for me at the moment, I prefer to let uv cook
- thangngoc89 2y agoI just take a look at the uv feature list and the feature I wanted most is Python version management. I'm using micromamba to install all interested Python versions (from 3.8 - 3.12) and set tell poetry about the python location. But like you said, poetry is working so well so I'll wait a little bit longer before jumping the ship.
- zbentley 2y agoRye wraps uv and adds python version management, among other things.
- xrd 2y agoSince you mentioned uv and the topic is virtual environments... I am using uv and it seems great. I don't understand the difference between using "python -m venv .venv; source .venv/bin/activate" and creating a venv with uv and then running the same source command. What does uv offer/integrate that's not present already in python venv module?
- zeotroph 2y agoAh, more complex than I thought: "venvstacks allows you to package Python applications and all their dependencies into a portable, deterministic format, without needing to include copies of these large Python frameworks in every application archive.", and in "Caveats and Limitations" (please, all projects should have one): "does NOT support combining arbitrary virtual environments with each other". Is there a helper to merge venv1 and venv2, or create venv2 which uses venv1 dependencies and on load both are merged?
- HappMacDonald 2y agoMy understanding is that the entire reason that venv exists is because python's library system is nothing but dependency spaghetti: whatever is needed by one project conflicts with whatever is needed by another so you have to give them bespoke library environments where those conflicts won't interfere with one another. In that perspective "merging" them directly defeats the purpose. What is needed is a better library ecosystem.
- zahlman 2y agovenvs are used to isolate groups of dependencies, but it's not just about conflicts. Many other languages expect you to do the same thing; people complain less because either the language statically resolves imports and can support multiple versions of a library in the same environment; and/or because the ecosystem has some conventions that allow the tooling to detect the "current environment" better, and/or because the standard installer isn't itself a library that defaults to appearing in every environment and installing specifically for "its own" environment; and/or (possibly the biggest one) they don't have to worry about building and installing complex multi-language projects where the one they're using is just providing a binding. An important reason for using them is to test deployment: if your code works in a venv that only has specific things installed (not just some default "sandbox" where you install everything), then you can be sure that you didn't forget to list a dependency in the metadata. Plus you can test with ranges of versions of your dependencies and confirm which ones your library will work with. They're also convenient for making neat, isolated installations for applications. Pipx wraps pip and venv to do this, and as I understand it there's similarly uvx for uv. This is largely about making sure you avoid "interference", but also about pinning specific versions of dependencies. It also lowers the bar somewhat for your users: they still need to have a basic idea of what Python is and know that they have a suitable version of Python installed, but you can tell them to install Pipx if they don't have it and run a single install command, instead of having to wrestle with both pip and venv and then also know how to access the venv's console scripts that might not have been put on PATH.
- nmstoker 2y agoI must be missing something because I understood that pip cached packages already. Perhaps it still creates a copy of the package files in the virtual environment, thus the cache only saves repeated downloads and not local disk space. If that's the case then this does look really useful.
- zahlman 2y agoPip caches downloads, but in the sort of filesystem database that git uses. You can't even directly pull .whl files (nor unzipped folders) out of it. Every virtual environment gets its own, unzipped copy of the wheels represented in that database. So yes, it's only "saving repeated downloads". (And it will still talk to PyPI by default to see if there's a newer version. There's an outstanding issue to add a proper "offline mode": https://github.com/pypa/pip/issues/8057 https://github.com/pypa/pip/issues/8057) It doesn't need to work like that. A lot of libraries would work fine directly from the wheel, because they're essentially renamed zip files and Python knows how to import from the archive contents directly. But this doesn't work if the package is supposed to come with any mutable data, nor if it tries to use ordinary file I/O for its immutable data (you're supposed to use a standard library helper for that, but awareness is poor and it's a hassle anyway). Long ago, the "egg" format expected you to set a "zip-safe" flag (via Setuptools, back when it was your actual packaging tool rather than just a behind-the-scenes helper wrapped in multiple backwards-compatibility layers) so that installers could choose to leave the archive zipped. But I don't know that Pip ever actually used that information, and it was easy to get wrong. But more importantly, the contents of the virtual environment could be referenced from a cache that contained actual wheels (packed or unpacked) by hard links, `.pth` files (with a slight performance hit) or symlinks (I'm pretty sure; haven't tested).
- yags 2y agoCouple more resources: Github Repo: https://github.com/lmstudio-ai/venvstacks https://github.com/lmstudio-ai/venvstacks Blog post: https://lmstudio.ai/blog/venvstacks https://lmstudio.ai/blog/venvstacks Docs: https://venvstacks.lmstudio.ai/ https://venvstacks.lmstudio.ai/
- dang 2y agoChanged to the blog post from https://pypi.org/project/venvstacks/ https://pypi.org/project/venvstacks/, since it gives more background. Thanks!
- Vaslo 2y agoWhy would this be an advantage to Astral’s uv?
- woadwarrior01 2y agoNone. FWIW, this exists because they wanted to ship a whole Python environment with an electron app.
- deleted 2y ago[deleted]
- ewuhic 2y agoPython venv tools are the new JS frameworks: every day some other ridiculous tool is born. And yes, everything is already solved with: due diligence (non-existent in scientific community) and nix.
- batch12 2y agoCan you clarify your second sentence? I'm having a hard time understanding the point you're making.
- ewuhic 2y agohttps://news.ycombinator.com/item?id=42032174 https://news.ycombinator.com/item?id=42032174
- angra_mainyu 2y agonix mentioned let's go In all seriousness, it is a bit tiresome when you come back to the python world and see a completely fragmented ecosystem (poetry, pdm, pip-tools, uv, and the older ways).
- zahlman 2y agoThese tools are attempting to solve different problems (really, different, overlapping sets of problems) - because there isn't even really agreement on what the problems are. My personal view is that there "one standard tool" makes sense for users (i.e. people who will install applications, install libraries in order to create their own personal projects, etc.) but not for developers (i.e. people who will publish their code, or need to care about rigorous testing, published documentation etc.). The latter require the users' install tool, plus specific-scoped tools according to their personal preferences and their personal conception of what the problems are in "development" that need solving. That is, they should be able to build their own toolchain, although they might start with a standard recommendation. Pip's scope has already crept in the sense that it exposes a `wheel` subcommand. It needs to be able to build wheels in order to install from sdists, but exposing this functionality means providing... half of a proper build frontend. (PyPA already offers a complete, simple, proper build frontend, called `build`; if `pip wheel` is part of your deployment process, you should probably be using `build` instead.)
- whinvik 2y agoBut can it do GDAL. (This is the Doom equivalent joke for Python environments!)
- wcrossbow 2y agoHere is one piece of knowledge earned the hard way. Do not use the official python GDAL bindings (`pip install GDAL`) unless you really must. Go with rasterio and don't look back.
- dagw 2y agoGo with rasterio and don't look back Excellent advice. Add fiona and shapely for manipulating vector data, and pyproj for projections and coordinate systems. Yes there are corner cases where installing 'real' GDAL is still needed, but for most common use cases you can avoid it.
- whinvik 2y agoYes, I have generally tried to do that. But sometimes there are third party libraries that depend on GDAL and then there's no way around it.
- superkuh 2y agovenv is admitting that there no longer is a Python language, only multitudes of pythons, one for each application you want to run.
- seamossfet 2y agoIt's crazy to me that in 2025 we still haven't figured out python dependency management
- kevin_thibedeau 2y agoDebian had it figured out, then they had to go and ruin it all by hopping on the venv bandwagon.
- biorach 2y agoIn fairness, this is a response to a need which became common only relatively recently
- fb03 2y agoWe figured, but people can't stand still and just Do Shit (TM). Poetry works fine and solves dependencies and peer dependencies. But I guess the javascript-style churn of 'every week a new tool which is better than the formers' has arrived too in Python land.
- vdvsvwvwvwvwv 2y agoPoetry in theory works fine but I found it a pain to use on Windows. As in flakey. A project works, then a dependency can't be found, so I reboot and then it can. The tools to manage and diagnose why it is not installing to the right env are non existant. I am probably not holding it right... but should I need to deep dive rabbit hole into installing dependencies. It is a non issue in NPM, .NET's Nuget, Haskell Stack, Ruby, etc.
- VertanaNinjai 2y agoWhat do you mean? Just venv your venv.
- pxc 2y agoIt would be nice if Python could actually support this. But 'venv detection' is conventionally (everywhere) done by a stupid, never-designed involving inspecting argv0. At the same time, most Python packaging tools make various assumptions like "if I'm in a venv, I can write into it", or "there will only ever be one level of venvery, so I'm not in a venv iff I'm looking at the system Python". Nesting/layering venvs has never been supported by any Python dependency management tool except for experiments that were rejected by more senior people in the community, the maintainers of major dependency management tools, etc. Same kind of thing goes for allowing multiple versions of a dependency in the dependency tree at runtime. There's little appetite for fixing Python's dependency management issues in a principled or general way, because that would require breaking things or adding limitations. You can't have dependencies partially managed by arbitrary code that runs only at install time and also build good packaging tools, for instance.
- marban 2y agoWhat's wrong with Pyenv & Poetry?
- dagw 2y agoNothing per se. But there are a few different workflows and project types they either don't support or make very difficult. If pyenv and poetry solve all your problems then it's a perfectly fine setup.
- v3ss0n 2y agoWhy not just use PDM
- skissane 2y agoI watch my CI pipeline spend a minute or two creating the virtualenv and pip install into it several times a day, despite the fact that nothing has changed (no change to requirements.txt, etc), and I wish there was a well-supported way to package up a venv into some kind of artifact (keyed by Python version, host platform and package versions) so we only had to rebuild it if something changed. This sounds a bit like that, but also sounds like something more complicated than what I was looking for.
- nerdbaggy 2y agoGitlab CI can do that. https://docs.gitlab.com/ee/ci/caching/#compute-the-cache-key-from-the-lock-file https://docs.gitlab.com/ee/ci/caching/#compute-the-cache-key...
- smetj 2y agoPut it in a container?
- zahlman 2y agoIs it re-solving for dependencies every time? Have you checked where the bottleneck is? (Don't CI pipeline tools hold onto Pip's download cache? Maybe this is part of why Setuptools gets so many downloads...)
- nsteel 2y agoI thought setup-python handled that already, no? https://github.com/actions/setup-python?tab=readme-ov-file#caching-packages-dependencies https://github.com/actions/setup-python?tab=readme-ov-file#c... There's also the pip-compatible mode of uv which is much, much faster than pip.
- nsteel 2y agoDoes this support non-python dependencies too e.g. gstreamer? It wasn't clear to me. I'm very wary of a new tool announcement that doesn't appear to mention why the existing tools/solutions were not sufficient. Which gap does this fill? Edit: the answer to my second question is at https://venvstacks.lmstudio.ai/design/ https://venvstacks.lmstudio.ai/design/