5 ms·
Classifying Python virtual environment workflows
- fake-name 4y agoOne option I never see mentioned in any of these discussions about virtualenvs/stuff is my approach to python environments: Each project gets it's own VM/Container. You then have one python version in the container, and no virtualenvs at all. All libraries get installed right into the main python libraries folder. I've been using this approach for 5+ years now, and I've yet to hit issues. I read people lamenting issues with python's packaging infrastructure, and with the approach I've never run into any of the issues they talk about. Pip complains a lot about being told to install stuff globally, but from what I can tell as long as you are careful enough to never use distro-sourced python packages (they're almost always out of date anyways, at least on ubuntu), it's never caused problems for me. For web-service type things, this has a lot of knock on effects that are beneficial. Each project winds up with it's own IP, if you need to restart things you can often just reboot the VM/container (assuming you have init scripts), process/library isolation issues are effectively impossible, etc...
- leksak 4y agoDo you store your requirements using pip freeze then?
- fake-name 4y agoYep.
- methou 4y agosame. but I'm still wishing for a native container support from pipenv/pyenv/poetry etc.
- mattbillenstein 4y agoSeparation of concerns...
- marban 4y agoI've settled with: Pyenv > python venv > Poetry
- thenipper 4y agoSame. Though I’m debating swapping out poetry with pip tools.
- Zizizizz 4y agoHave you used pip-tools before? I have used PDM instead of Poetry and was very impressed https://github.com/pdm-project/pdm https://github.com/pdm-project/pdm
- jessekv 4y agoFor what its worth, I evaluated pdm, poetry, and pyenv before settling on pip-tools for my workflow. pdm is the only one I would consider trying again (once it is more mature).
- thenipper 4y agoI have. At my last gig at Shopify we used it to manage our requirements since poetry didn’t play nice with some internal dev tools. I’ve tried PDM as well. It’s definitely nice. Part of me just wants to avoid “all encompassing” tools and use a specific tool for a specific job.
- tomwojcik 4y agoI started using poetry for my OSS projects and I'm happy with it, but I'd never use it for closed source projects. Poetry simplifies deps, config, pkg build etc as it's stored in a single place, but it's way to opinionated. For managing dependencies only, pip tools is easier to use and customize. There's no breaking changes, it's a finished product. Also, I don't want my CI to fail randomly. https://github.com/python-poetry/poetry/pull/6297 https://github.com/python-poetry/poetry/pull/6297 Look at the number of issues in poetry. Some things are not backwards compat, some are broken. If you want reliable, KISS with pip tools. That's my opinion.
- molly0 4y agoOnly reason I use virtual environments instead of just building everything inside Docker containers is the debug experience, which is not good enough inside a Docker container.
- TekMol 4y agoHow is debugging inside a container different than on the host os?
- molly0 4y agoWell to use something like "debugpy" It's the port mapping that just dosn't work straight out of the box, but needs custom settings inside my editor (and often times additional code inside the startup script) to get it working between the container and my local machine.
- TekMol 4y agoIt sounds like you want to do the debugging outside of the container? I just do it inside the container with pudb.
- Zizizizz 4y agoHow does your LSP pick up everything like imports if they only live in a docker container?
- TekMol 4y agoAre virtual environments still needed now that we have containers? I just run everything in a container and never need a virtual environment. Am I missing out on anything?
- molly0 4y agoFor me it's the debug experience that is lacking when only using containers, I have some work arounds but they are still not good enough.
- TekMol 4y agoCan you give an example? I don't experience any difference, whether I am inside a container or on the host os.
- pnt12 4y agoCan you debug in a container, eg VS code debugger with breakpoints and steps?
- TekMol 4y agoI have no experience with that as I use pudb for debugging, which I simply run inside the container.
- 0x008 4y agoYea it is a supported use case. VS code remote extension can work with every container just like remote ssh development.
- crucialfelix 4y agoyes, devcontainers work very well. Full debugger. It's also possible to run the container in the cloud and still get a full local debugger. I did have some issues with some projects and want to revisit this soon.
- tsbischof 4y ago
- IanOzsvald 4y agoCan other data scientists comment? I'm 15 years in with python and scientific work. For a lot of years I liked conda but then it got crazy slow. Next I started making conda environments and installing packages with pip. Now I'm experimenting with mamba ("fast conda") and that's pretty good. Conda envs mean I can experiment with different versions of Python (I'm a co author for O'Reilly's High Performance Python so eg 3.11 and 3.12 are pretty interesting right now). Conda "should" also make identical teaching environments (I teach my own courses). Pip was a pragmatic choice to get installations in minutes not hours in the years when conda was silly-slow. The above is also all for short -lived research work (my typical client mode for scientific work), so it is probably different to anyone doing long-run dev work, production deploys, or for those not needing non-Python binary support (eg GPU/C/Fortan lib support).
- longevyti 4y ago> Now I'm experimenting with mamba ("fast conda") and that's pretty good. +1 for this. I really like Conda because I use a lot of packages that cannot be installed with only pip (E.g. GDAL). Conda would sometimes take 10 minutes to make an environment. Mamba has been an absolute game changer. It has rewritten parts of Conda in C++ and added multithreading, so every part of the process - the solving, download, and the extraction - is all Lightning quick in comparison. A 10 minute process with Conda can now take 2.5 mins with Mamba. It also helps that it is also prettier to look at!
- plonk 4y agoI found that the best way to install GDAL was: - on Windows, using Christoph Gohlke’s wheels, - on Linux, installing GDAL with the OS’ package manager, then the Python bindings by fixing the version to $(gdal-config —-version) and setting C_INCLUDE_DIR and CPP_INCLUDE_DIR. It’s a bit more involved, but the speed of pip and conda’s multiple channels and dependency resolution issues make this worth it.
- chazeon 4y agoAgreed. I recently saw on a Mamba team's blog post saying that conda is also starting to incorporating libmamba, so probably the issue could be resolved someday.
- Zizizizz 4y agoMy flow that I have settled on as being the most predicable (even though PDM allows for installing to a local virtualenv as well) is (with pyenv): mkdir newproject && cd newproject echo 3.11.0 > .python-version # (If different python version) python -m venv .venv # Make the venv And I have this in my `.zshrc` so whenever I `cd` to that directory, it auto-activates the virtual environment. Having the local directory in the project also makes it really easy for my editor (neovim or VS Code) to pick up export PYENV_ROOT="$HOME/.pyenv" command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH" # Custom key-bindings python_venv() { MYVENV=./.venv # when you cd into a folder that contains $MYVENV [[ -d $MYVENV ]] && [[ ! $VIRTUAL_ENV ]] && source $MYVENV/bin/activate # when you cd into a folder that doesn't [[ ! -d $MYVENV ]] && [[ $VIRTUAL_ENV ]] && deactivate } autoload -U add-zsh-hook add-zsh-hook chpwd python_venv python_venv
- truculent 4y agoDo you have any issues with many virtualenvs/packages taking up a lot of disk space? For instance, numerical packages (scipy, torch, etc) can add up if they’re duplicated across many projects, and, as I understand it, virtualenv doesn’t do any caching/symlinking.
- Zizizizz 4y agoI don't work on too many projects at a time and tend to archive and remove the virtualenv when I finish working on something for the foreseeable future. I can see why that would be a problem for some though.
- fbdab103 4y ago>...virtualenv doesn’t do any caching/symlinking. Well, one of us is incorrect then. I had been thinking for years that one of the key benefits was that virtualenv would not do a blind copy of files (except on Windows...sigh). The `venv` command even has a `symlinks` vs `copies` option. `python3 -m venv --help`
- truculent 4y ago
- computerfriend 4y agoI hope the convention becomes to also consider ./venv as another default location.
- brundolf 4y agoThinking about how python is the only ecosystem where you need to deal with any of this, and how many thousands of developer-hours are spent on it
- arturkane7 4y agoGood point. I am wondering what are the specific reasons say Java or JS don’t suffer from this, and hoping for rant-free responses that don’t turn into language wars.
- IgorPartola 4y agoIn part because Python is often used as an OS language (Debian has it in the base system and a bunch of OS subsystems use it) so if you upgrade some library at the OS level it affects more than just your own code. In part it is because Python has modules, not libraries. Modules can have executables bundled with them. In part, it’s because Python modules packages don’t exactly care about backwards compatibility. By and large, people who put out packages will keep a changelog but not keep the same API going for years. So slightly different version of the same package can give you different results. Lastly it is because Python does not have a clear definition of a project. If at the language level I could say “this is the root of my project. Do not use any OS-level libraries. Here are my library dependencies and they are to be stores in $PROJECT_ROOT/libs/“ then none of this would be required. I have rarely found that the version of Python itself was a big problem for me, but having the ability for multiple versions to be installed at the OS level combined with the above mechanism would entirely eliminate the need for virtual envs.
- nonethewiser 4y agoFor JS, which I'm intepreti as node because I think that's the real parallel, the answer is NVN and NPM. Or alternatives such as yarn or pnpm. But even the default package manager is great compared to python. I havent tried it but my understanding is poetry is a bit more like npm.
- Kwpolska 4y agoI would say that one important factor is the tooling. Instead of having one or two standard tools, as seen in Java (maven/gradle), Node (npm/yarn), .NET (dotnet CLI and MSBuild/maybe FAKE), Python has a plethora of opinionated tools. venv+pip is built-in, but doesn’t handle any of the “where to put the venv” parts, and doesn’t handle making packages installable (and the classic solution is setuptools). There are many competing projects, like pipenv (which does not work at all for libraries), poetry (which uses a non-standard way of specifying metadata and dependencies, and isn’t too friendly with the rest of the packaging world), flit, hatch, etc. Why are there so many competing tools? Why can’t the community just pick one and make it a good tool for all use-cases? There’s an organization called “Python Packaging Authority”, and it maintains almost all tools mentioned in the previous paragraph (except the stdlib venv, and poetry). If they’re an authority, they should just say “___ is the way to go, we’ll add the missing use-cases to it, here’s a tool to migrate everything else to ___”. Instead, they have done things that support the proliferation of tools, like the PEP 517 (which is a standard API for package managers to talk to build tools). Another deficiency in the Python package management system is the fact that you need virtual environments to get things done, and that they are often finicky. Some of the modern tools hide the venv from you, but this gets less practical if you’re trying to use things from within scripts, or trying to point your WSGI server at the venv, or in multi-user scenarios. Some of the modern tools put it in .venv and manage it for you, but venvs can randomly break in cases the tool might not be aware of (eg. in some system package managers, upgrading Python to a new minor version would cause venvs to break, because symlink targets moved.) Node solves this by having a `node_modules` directory. .NET does stuff at build-time that most people don’t need to care about, and then just puts the .dll files for your dependencies next to the .dll with your thing, and the loader looks in the directory your code came from. Python has a proposal for `__pypackages__` [0], but it’s been there since May 2018 with not much progress. [0] https://peps.python.org/pep-0582/ https://peps.python.org/pep-0582/
- fbdab103 4y agoLooking for a suggestion from the audience here. I lean on Makefile for automation wherever possible, even if it is just a centralized trigger command to run a packaged script (eg scripts/dothething.py). One of my sticking points is how to ensure the Python virtual environment is installed and up to date before triggering any scripts. That is, I clone a new project, cd into it, and want to run some package command inside the expected environment (eg `poetry run scripts/dothething.py`). This will fail if the environment has not previously installed all of the defined dependencies. My annoyance is that requiring `poetry install` to run prior to every command is quite slow, and rarely has anything to do. Does anyone have a suggestion as to how to ensure the right thing is always done without the slow Poetry resolution? My dream is that in all of my projects you could, say `make test` and Python would optionally create the environment, install the dependencies, and run pytest in the fresh case. The typical development case would confirm that the environment exists and just run pytest.
- 323 4y agoYou can check that a venv exists by looking for .venv/pyvenv.cfg. Is that not enough? Unless you also need to check that all packages are installed and up to date.
- fbdab103 4y ago1) Confirming that the environment is in sync with my definition file (pyproject.toml or lockfile) is my true objective. If I suddenly add libraryFoo to pyproject.toml, but I forget to run `poetry install`, I want my `make test` command to note the discrepancy and correct the situation. 2) I think that has some brittleness of detecting where the environment lives (as per the author's post). For my use case, I would be fine assuming the defaults, but it is something else to go wrong Edit: Thinking out loud, I guess the quickest solution is to checksum the pyproject.toml file and store that somewhere. If current hash matches previous, do not run `poetry install`. While that would get me there, I am hoping for a first-party, standard solution.
- 323 4y ago
- marius_k 4y agoMy flow/boilerplate for apps: ./scripts/create-venv.sh # create venv using specific python version, update pip, wheel, setuptools, install requirements.txt ./scripts/freeze-requirements.sh # create tmp venv with requirements-base.txt and pip freeze > requirements.txt ./requirements-base.txt # dependencies with loose versions ./requirements.txt # frozen/locked dependencies
- chazeon 4y agoFor scientific calculation I just use mamba (sometimes micromamba) + managed environment in home folder. It is impossible to keep an environment for each project due to the library size and storage limitation. These I keep one copy each machine (including on HPCs and local machines). For bootstraping, I keep a copy of the environment.yaml file. I used to have a dedicated folder to manage these environments myself, but seeing conda/mamba does a good job managing these; and they are also easier to load with command, so I switched to the managed approach. Conda does not only work for Python, but many software necessary for our workflow that are written in C/C++/Fortran has been precompiled and uploaded to the conda-forge index as well.