23 ms·

How Python virtual environments work

• Max_Limelihood 4y ago
  Answer: they don’t (Seriously, I’ve gotten so fed up with Python package management that I just use CondaPkg.jl, which uses Julia’s package manager to take care of Python packages. It is just so much cleaner and easier to use than anything in Python.)
  • swyx 4y ago
    also there's like 3 different flavors of virtual env now and me being 8 years out of date with my python skillz i have no idea what the current SOTA is with python venv tooling :/ i dont need them demystified, i need someone smarter than me to just tell me what to do lol
    • ollien 4y ago
      The reality is that if you ask 3 different people you're going to get 3 different answers. They're fundamentally the same, just a matter of package management. As far as I'm aware, the current "SOTA" is Poetry. I liked Pipenv for quite some time, but Poetry is just so much faster IME.
    • gosukiwi 4y ago
      It also makes it very hard for new devs willing to learn Python. Coming from Ruby and JavaScript, you just use bundler or npm, but Python is so strange, even the way it runs files is different, with the module thing.
    • dwringer 4y ago
      > and me being 8 years out of date with my python skillz i have no idea what the current SOTA is with python venv tooling :/ It doesn't really matter, by the time you sit down and use it you'll find whatever that is, has also been deprecated and replaced by 2 more.
      • billforsternz 4y ago
        The problem with software development in 2023 in a nutshell, well played sir.
    • hot_gril 4y ago
      > i dont need them demystified, i need someone smarter than me to just tell me what to do lol Dockerfile ;)
  • dangerlibrary 4y ago
    I hate python package management - I really do. But I've never actually had a problem with virtual environments, and I think it's because I just use virtualenv directly (rather than conda or whatever else). I have these aliases in my .bashrc, and I can't remember the last time I had a major issue. alias venv='rm -rf ./venv && virtualenv venv && source ./venv/bin/activate' alias vact='source ./venv/bin/activate' alias pinstall='source ./venv/bin/activate && pip install . && pip install -r ./requirements.txt && pip install ./test_requirements.txt' I don't have all the fancy features, like automatically activating the virtualenv when I cd into the directory, but I've always found those to be a bigger headache than they are worth. And if I ever run into some incompatibility or duplicate library or something, I blow away the old venv and start fresh. It's a good excuse to get up and make a cup of tea.
    • Izkata 4y ago
      > virtualenv venv That would be python2, in 3 it's "python -m venv venv" (first venv is package to run, second is directory to put it in) Otherwise yeah, it's the same and I also use it manually. Never had any problems.
      • jvolkman 4y ago
        `virtualenv` still exists and is still actively developed. It's true that Python 3 ships with `venv` but I think `virtualenv` offers some additional features. https://github.com/pypa/virtualenv https://github.com/pypa/virtualenv
    • jimnotgym 4y ago
      Agreed. I tried the new package manager combined with venv and using venv directly seems best. A lot faster for a start.
    • spprashant 4y ago
      I might steal these aliases, thank you. Using virtualenv directly has also been my approach, and has not failed me yet. I also used Poetry for one of my personal projects, and I liked what I saw.
    • 4y ago
  • birdstheword5 4y ago
    It sounds mean to say it, but it's 100% true. I moved away from using python wherever I can. I've had colleagues struggle for days to install well used packages like pandas and numpy in conda.
    • danielvaughn 4y ago
      I just began writing Python a few months ago. For years prior, I'd been a JS dev, and while NPM can be frustrating at times, I never encountered so many issues as I have in Python. It's crazy. I'm now curious whether there are languages out there that do have a really nice packaging system.
      • dvlsg 4y ago
        Cargo (Rust) is pretty solid. Most of my minor complaints (like being unable to add packages from the CLI) have been resolved with time, as well.
      • Nadya 4y ago
        Personally, zero complaints about Cargo (Rust) and very minimal complaints about NuGet (C#/.NET). My issues around NuGet are probably self-created because I refuse to learn the CLI [0] for it and I've had occasional issues with Visual Studio's UI for managing things. https://learn.microsoft.com/en-us/nuget/reference/nuget-exe-cli-reference https://learn.microsoft.com/en-us/nuget/reference/nuget-exe-...
        • evntdrvn 4y ago
          In a lot of ways, Paket is significantly better than NuGet, if you ever want to try something new :) It uses a lockfile approach like Cargo, has better dependency resolution, etc https://fsprojects.github.io/Paket/index.html https://fsprojects.github.io/Paket/index.html
      • cstrahan 4y ago
        FWIW, I find Cargo to be one of the biggest reasons I like Rust so much — maybe even more than anything to do with Rust itself or safe code. I’ll often look for command line tools written in Rust, but not because of Rust fanboyism, but because I know I can just git clone the project and immediately start hacking on a new feature I need or a quick bug fix. In almost every other language I have to jump through one million hoops before I can build and run whatever it is, let alone have a nice developer experience (autocomplete, go to definition, etc).
• 89vision 4y ago
  I haven't used these since docker
  • foooobaba 4y ago
    With docker, do you use debugging in pycharm/vscode, or just for compiling/shipping?
    • 89vision 4y ago
      Both. Setting up the editor took a little doing, but it works well. https://code.visualstudio.com/docs/containers/quickstart-python https://code.visualstudio.com/docs/containers/quickstart-pyt...
  • ginko 4y ago
    That's just giving up.
    • chao- 4y ago
      Yes it is giving up, but not only. It is giving up and being able to get back to the actual work you want to be doing.
      • PaulHoule 4y ago
        I worked at a place where people tried using Docker to manage Python and wound up with a bunch of twisty little images that were all misconfigured in a different way. (E.g. default charsets that I’m not sure anybody uses.)
        • cpach 4y ago
          Learning to make sound container images is tricky, but once you got it figured out it is, IMO, a very nice way to distribute software.
    • frgtpsswrdlame 4y ago
• cmcconomy 4y ago
  My personal approach is: - use miniconda ONLY to create a folder structure to store packages and to specify a version of python (3.10 for example) - use jazzband/pip-tools' "pip-compile" to create a frozen/pinned manifest for all my dependencies - use pip install to actually install libraries (keeping things stock standard here) - wrap all the above in a Makefile so I am spared remembering all the esoteric commands I need to pull this all together in practice, this means once I have a project together I am: - activating a conda environment - occasionally using 'make update' from to invoke pip-compile (adding new libraries or upgrading), and - otherwise using 'make install' to install a known working dependency list.
  • nose-wuzzy-pad 4y ago
    This seems simplistic and low drag. Do you have an example you can share? Thanks!
    • cmcconomy 4y ago
      Sure: https://gist.github.com/cmcconomy/fa9cad3fda009e522264ea8a21a9a811 https://gist.github.com/cmcconomy/fa9cad3fda009e522264ea8a21... https://gist.github.com/cmcconomy/9bca20856a6a48704555bc8dcf169ba7 https://gist.github.com/cmcconomy/9bca20856a6a48704555bc8dcf... Hope this helps!
  • harold_b 4y ago
    [dead]
  • raihansaputra 4y ago
    thanks for sharing. I've thought about the same approach. Conda installs are.. annoying to say the least, but they do provide a better UX compared to manually managing venvs. Your approach seems mature. (why not ./venv/ per project? because you can't do that when your project directory is on another disk) (also i got burned with poetry in regards of very long dependency checking. I'm not making libraries, just an environment for my own projects)
• gt565k 4y ago
  Just setup a django project with pipenv, works just fine.
  • aniforprez 4y ago
    Pipenv has never once worked just fine personally. The dependency resolution is a joke and the slowest of any project in this space, they have tons of bugs and the project is languishing I prefer to use a combination of pip-tools and pyenv for my projects
    • zelphirkalt 4y ago
      There was a time, when pipenv seemed to be the most precise in dependency constraints resolution out of the tools available. Poetry did not see some constraints iirc, and pip did not check at all. However Poetry has developed much faster than pipenv and pipenv breaks too often and is left far behind by Poetry now.
• cozzyd 4y ago
  The "global" vs. "directory" dichotomy seems... off. Haven't PYTHONHOME and PYTHONPATH been supported since approximately forever?
• PhysicalNomad 4y ago
  I don't bother with venvs anymore and just use podman instead.
• sakex 4y ago
  It feels like it is one of the reasons experienced devs are ditching Python for production systems. Besides horrendous performance and lousy semantics. The cost of setting up, maintaining the environment and onboarding people is just not worth it.
  • lockhouse 4y ago
    Excuse my ignorance, but aren’t Virtual Environments something you setup once per project? Why would that be a dealbreaker? How is it any more difficult than putting everything in a docker container like all the cool kids are doing these days?
    • tmpz22 4y ago
      * Local state is changing such as brew updates or new dependencies are added. * External state is changing such as project contributions So its not a one-off unless the project and dev environment is static. The real problem is different tooling doing different amounts of hand holding and automation. Your editor may configure some things automatically, brew may configure some things automatically, and so a set of instructions for setup or problem fixing could be voided without the end user knowing. So now you're off on a adventure of unknown duration wading through internet forums trying to find the resolution that works for you. Ironically using Docker to isolate the environmental changes is a approach some people use to avoid some of this esoteric crap.
      • lockhouse 4y ago
        Isn’t it considered best practice not to use brew to install Python for exactly this reason? I’ve always seen it recommended to use pyenv or just download directly from Python.org instead.
        • PaulHoule 4y ago
          Python has always struggled with distribution maintainers that just don’t get it. Python barely survived becoming the default scripting language on Red Hat and other Linux distros which was a major obstruction to the Python 3 transition. If the new cohort of pandas and scikit-learn users had not been such a force of nature we”d be talking today about Python the way we do about Perl. Not installing venv is a serious crime on the part of Debian as beginners don’t need any more excuse to trash their system with site-local directories and other wrong answers for how to manage packages in Python.
• tomalaci 4y ago
  I would highly recommend Poetry for python package management. It basically wraps around pip and venvs offering a lot of convenience features (managing packages, do dist builds, etc.). It also works pretty nicely with Tox. I would recommend using virualenvs.in-project setting so Poetry generates venv in the project folder and not in some temporary user folder.
  • winrid 4y ago
    Thanks. I recently spent a whole afternoon learning how to package a new python project. Was really surprised at the difficulty even with venv, compared to node and java.
  • nerdponx 4y ago
    I prefer Hatch over Poetry. I don't have any strong reason for that preference, I've just use both and I feel more comfortable with Hatch. It feels a little more seamlessly integrated with other Python tools, and I appreciate the developers' conservative approach to adding features.
  • davidktr 4y ago
    100% this. I've always struggled with creating packages, but now simply do poetry init and I am done. Magic.
  • peterhil 4y ago
    I just compared and evaluated Hatch, Flit, Poetry and Pdm and found Pdm to be most robust and slimmest. Hatch was a good second option, and Poetry and Hatch are easy to use, but have too much bloat and magic.
    • pnt12 4y ago
      When I tried pdm it wasn't stable yet and messed up my paths. My experience with poetry has been great. I only disliked that they had auto-update on when locking files, but they changed the default.
• Havoc 4y ago
  These days I'm just throwing each project into a fresh LXC on a server. All these different languages have their own approach and each then also user/global/multiple versions...it's just not worth figuring out
  • spyremeown 4y ago
    Question: what makes you choose LXC over Docker?
    • wyufro 4y ago
      IMO, while their use cases do overlap, LXC is more geared towards a user installing things while Docker is more geared towards a developer creating a ready to use package. LXC creates environments, while Docker creates apps, is another way to say it.
    • Havoc 4y ago
      Much of a sameness really but I prefer the more persistent disk style of lxc plus ssh plus vscode ssh remote extension. Depends on task though I've got dockers and VMs in use too
  • bagels 4y ago
    Same, but with Docker. I don't like conda, was fine with virtualenv, but using docker there's only one python and you can just pip install and not worry about multiple environments.
• buildbot 4y ago
  I personally hate Conda with a firey passion - it does so much weird magic and ends up breaking things in non obvious ways. Python works best when you keep it really simple. Just a python -m venv per project, a requirements.txt, and you will basically never have issues.
  • bobbylarrybobby 4y ago
    If it were really that simple, surely all these other solutions wouldn't exist?
    • buildbot 4y ago
      Not saying Conda et. al don't solve problems, especially in specific cases, but they also add them in my opinion.
    • Spivak 4y ago
      So there's two notions of simple. Simple in the sense that it's actually simple, the software you need can be installed with pip install with precompiled binaries for your platform when necessary it supports Python 3.something+, and all it's dependencies are either >= version or the version is >= x.y <= x+1.0 Then there's simple as in the software is actually incredibly useful but is an absolute nightmare of complicated dependency trees where only specific pinned minor versions work together, you need multiple incompatible compiler toolchains and distro packages, it only works if you have CUDA, precompiled binaries exist for some but not all and if you use the precompiled binaries then it changes the dependency story, if you want jupyter support that's a whole different thing AHHHHHHHHHHHHH In that case some people with more time than sanity said fuck it we'll make it work and conda was born. For me it's a lifesaver when you want to use a piece of software but I wouldn't ever dare deploy production software with it without it being strongly isolated from everything else.
    • zzzeek 4y ago
      it really is that simple conda exists because it deals with an entirely different package registry and is a whole distro on its own (I dont know why people need that either, my vague impression is that scence-y types want complete pushbutton installation, typing a command == fail, I guess, I dont know). poetry exists because it does some kind of automated version management thing (as did pipenv), that I'm sure is nice but not something I've ever needed, but the buzz got out and it took over, people who have never typed "virtualenv" use poetry and they have no idea why.
• _coveredInBees 4y ago
  I'm surprised at the number of people here complaining about venvs in Python. There are lots of warts when it comes to package management in Python, but the built-in venv support has been rock solid in Python 3 for a long time now. Most of the complaints here ironically are from people using a bunch of tooling in lieu of, or as a replacement for vanilla python venvs and then hitting issues associated with those tools. We've been using vanilla python venvs across our company for many years now, and in all our CI/CD pipelines and have had zero issues on the venv side of things. And this is while using libraries like numpy, scipy, torch/torchvision, etc.
  • aflag 4y ago
    It's incredibly lacking in features. PyPI doesn't even properly index packages, making pip go into this dependency resolution he'll trying to find a set of versions that will work for you. It works for simple cases with few dependencies/not a lot of pinning. But if your needs are a bit more complex it certainly shows its rough edges. I actually find it amazing that they python community puts up with that. But I suppose fixing it is not that pressing now the language is widely adopted. It's not going to be anyone's priority to mess with that. It's high risk low rewards sort of project.
    • Groxx 4y ago
      What does that have to do with venvs? I agree the packaging and distribution setup in python is an absolute mess, but that's entirely unrelated to venvs. It's like bringing up how python uses whitespace instead of curly-braces.
      • oneepic 4y ago
        I think the GP comment might have caused some confusion since it mentioned both package management and venvs very close together.
      • deleted 4y ago
        [deleted]
      • 4y ago
• its_over_ 4y ago
  I use poetry or docker or nixpkgs I've given up. EDIT: also just finding myself reaching for go in most cases
• hkgjjgjfjfjfjf 4y ago
  [dead]
• josteink 4y ago
  All other languages: use whatever packages you like. You’ll be fine. Python: we’re going to force all packages from all projects and repos to be installed in a shared global environment, but since nobody actually wants that we will allow you to circumvent that by creating “virtual” environments you can maintain and have to deal with instead. Also remember to activate it before starting your editor or else lulz. And don’t use the same editor instance for multiple projects. Are you crazy??? Also: Python “just works”, unlike all those other silly languages. Somebody IMO needs to get off their high horse. I can’t believe Python users are defending this nonsense for real. This must be a severe case of Stockholm-syndrome.
  • rad_gruchalski 4y ago
    Yeah, how does it go? There’s at least one obvious way to do something? Python… makes me anxious. I don’t mind writing Python but setting it up is crazy.
  • frou_dh 4y ago
    Python is over 30 years old so it's hardly surprising that it's got plenty baggage in at least some areas.
  • analog31 4y ago
    I'm a long time Python user, and I don't defend virtual environments at all. I just don't use them. Granted, I'm a so called "scientific" programmer, and am not writing production code. I haven't run into the problems that are solved by virtual environments, nor have my colleagues. Sure, it means I'm probably living in a bubble, but it may be a bubble shared by a lot of people. Python is the first language that I've used, where the user community is a major attraction, resulting in significant inertia. Replacing Python requires a new language and a new community. Also, the tools that helped build that community, such as Google and Stackoverflow, have (by some accounts) deteriorated. If package management is that bad, then yeah, time to switch languages. No high horse here. Like Sancho Panza, I have to be content with an ass. ;-)
  • mixmastamyk 4y ago
    Basically the point is to avoid the system python, which is not hard. One needs some sys-ad skills to understand what is going on however; unfortunately sounds like they are short supply. I don't do anything you mention, so there must be a simpler way.
• Already__Taken 4y ago
  Been really enjoying trying out pdm in PEP 582 mode. I've just found it behaves when used across multiple devs, not necessarily that used to working with python.
• ggm 4y ago
  How much of this is caused by a join over "odd" decisions of what is installed by Python3 developers, "odd" decisions of what a "package" is by package makers and what I think I want to call "fanaticism" by Debian apt around things? FreeBSD ports are significantly closer to "what the repo has, localized" where it feels like linux apt/yum/flat is "what we think is the most convenient thing to bodge up from the base repo, but with our special sauce because <reasons>"
• killjoywashere 4y ago
  I didn’t realize venv was part of the standard library. If that’s the case, how is it that conda even exists? Anybody got a good history of this?
  • randoglando 4y ago
    venv is part of the standard library from Python 3. It's not in Python 2.
  • int_19h 4y ago
    conda can install things other than Python packages. C++ compilers, for example, or native libraries that Python packages depend on.
• deleted 4y ago
  [deleted]
• jcparkyn 4y ago
  I'm beginning to feel like every single comment in every thread related to python package management is just this: "Package management in python is so easy, just use [insert tool or workflow that's different to literally every other comment in the thread]."
• cpburns2009 4y ago
  Virtual environments are easy to create and manage. Create one with the built-in venv module: python3.10 -m venv ./venv # or your favorite version . ./venv/bin/activate pip install pip-tools Manage dependencies using pip-compile from pip-tools. Store direct dependencies in "requirements.in", and "freeze" all dependencies in "requirements.txt" for deployment: . ./venv/bin/activate pip-compile -U -o ./requirements.txt ./requirements.in pip install -r ./requirements.txt
• rekahrv 4y ago
  That's insightful. It seems that a virtual environment created by Poetry looks very similar, except that it doesn't contain an `include` directory. It contains: * `bin` directory * `lib/<python-version>/site-packages/` directory * `pyvenv.cfg`
• asicsp 4y ago
  See also: Virtual Environments Demystified (https://meribold.org/python/2018/02/13/virtual-environments-9487/ https://meribold.org/python/2018/02/13/virtual-environments-...) Discussion from 2021: https://news.ycombinator.com/item?id=25611307 https://news.ycombinator.com/item?id=25611307
• Supermancho 4y ago
  This writeup needs work. > So while you could install everything into the same directory as your own code (which you did, and thus didn't use src directory layouts for simplicity), there wasn't a way to install different wheels for each Python interpreter you had on your machine so you could have multiple environments per project (I'm glossing over the fact that back in my the day you also didn't have wheels or editable installs). This is a single run-on sentence. Someone reading this, probably doesn't know what "wheels" means. If you are going to discount it anyway, why bring it up? > Enter virtual environments. Suddenly you had a way to install projects as a group that was tied to a specific Python interpreter I thought we were talking about dependencies? So is it just the interpreter or both or is there a typo? > conda environments I have no idea what those are. Do I care? Since the author is making a subtle distinction, reading about them might get me confused, so I've encountered another thing to skip over. > As a running example, I'm going to assume you ran the command py -m venv --without-pip .venv in some directory on a Unix-based OS (you can substitute py with whatever Python interpreter you want Wat? I don't know what venvs are. Can you maybe expand without throwing multi-arg commands at me? Maybe add this as a reference note, rather than inlining it into the information. Another thing to skip over. > For simplicity I'm going to focus on the Unix case and not cover Windows in depth. Don't cover Windows at all. Make a promise to maintain a separate doc in the future and get this one right first. > (i.e. within .venv): This is where you start. A virtual environment is a directory, with a purpose, which is baked into the ecosystem. Layout the purpose. Map the structure to those purposes. Dive into exceptional cases. Talk about how to create it and use it in a project. Talk about integrations and how these help speed up development. I also skipped the plug for the mircoenv project, at the end with a reference to VSCode.
  • ianbutler 4y ago
    I expect most everyday python users know what these things are. I also expect this was targeted at python users who use these things but haven't thought deeply about them. Charitably, I will assume you are a non python user, and that's why this is a miss for you.
• warner25 4y ago
  > One point I would like to make is how virtual environments are designed to be disposable and not relocatable. Is the author saying that relocating them will actually break things, or that it's just as easy to recreate them in a different location? Because I've moved my venv directories and everything still seemed to work OK. Did I just get lucky?
  • Falell 4y ago
    Relocating them will actually break things in many cases, especially when native code is involved.
  • noisenotsignal 4y ago
    There’s also relocating across machines. For example, maybe your build environment has access to internal registries but your release environment does not. I naively thought you could build your venv and just copy to the new machine (both environments were Ubuntu) but ran into errors (due to links breaking). We also used pex for a bit, which is kind of like building a binary of a venv, and that eventually stopped working too when the C ABI was no longer the same between environments. There didn’t seem to be an easy way to pick the ABI version to target when creating the pex file, so I gave up and just downloaded the wheels for internal packages in the build.
  • acomjean 4y ago
    I’ve tried to move things and broken everything. (Conda environments). I tried replacing the paths in the files and it didn’t work. We run a bunch of different tools with various python requirements and would like to be able to duplicate them for the next tool. We ended up making a new environments for each. Honestly it’s a bit of a mess.
  • jszymborski 4y ago
    It's a gamble to move venvs. The real way to move venvs is to freeze the venv (i.e. make a requirements.txt) and then pip -r requirements.txt to recreate the venv. This process is really the only thing about venvs that ever causes me trouble.
  • Helmut10001 4y ago