23 ms·

You don't really need a virtualenv

169 points by gilad 6y ago

michaericalribo 6y ago
> One day a minor release of Python is out It is decidedly not true that I want to update my venv for every minor version bump! I deploy to a cloud service w/ a specific version; my package manager is slow to update; I develop collaboratively using a shared container w/ a fixed version. Updating shared resources with every new release is not always realistic, and that makes it so that I do need (or at least want) to use a virtualenv.
- dvfjsdhgfv 6y ago
  > It is decidedly not true that I want to update my venv for every minor version bump! I see your point, but testing against every minor version is a great way to spot potential problems (that you'll need to deal with anyway).
- swiftcoder 6y ago
  > It is decidedly not true that I want to update my venv for every minor version bump! Until you run into a minor version bump that patches a critical security vulnerability, and then you suddenly may very much want to (and in some corporate environments, have a regulatory requirement to)
  - michaericalribo 6y ago
    That’s not every minor bump, though, and is worth the extra effort! Also, my use case is 100% non critical from a security standpoint, so I can afford to be careless... your point definitely stands in sensitive environments
neolog 6y ago
Sure you can make your own package directory and develop a new package manager to point into it. But what's the point when Poetry already exists?
- cutler 6y ago
  As the authors said - no more virtual envs.
  - danmur 6y ago
    A virtualenv is just a bunch of packages, you end up with the same thing whatever you want to call it. It is probably a more roundabout solution that it needs to be, but I don't find many problems myself (also using poetry).
    - cutler 6y ago
      He also said virtual envs install their own copy of Python. No need for that either. Coming from other languages the whole virtual env thing seems alien especially the way it messes with your shell.
      - iandinwoodie 6y ago
        A virtualenv (venv) does not install its own copy of Python. Instead, when you create the venv it puts symlinks for the calling python in the <venv>/bin directory. When you activate the venv the <venv>/bin is added to the front of your PATH so that any "python" commands use the symlink'ed version of Python corresponding to your venv.
        aequitas 6y ago
        Indeed, a virtual env is really not that exciting or complicated. Even "activating" a virtualenv is just changing your shell's PATH so it picks the virtualenv binaries, nothing else. You can also just call those binaries directly from their installed location in the env without ever having to "activate" or "workon" anything.
      - danmur 6y ago
        Activating the venv is completely optional. I never do it, but some people seem to like it.
darkteflon 6y ago
This was a good article and looks like an interesting project. Regardless of whether or not you choose to _deploy_ with Docker, _developing_ in Docker containers using the VS Code Remote extensions really solved all of Python's (and Javascript's) annoying packaging problems for me, with the bonus that you also get to specify any additional (non-Python) dependencies right there in the repo Dockerfile and have the development environment reproducible across machines and between developers. YMMV, of course, but I find this setup an order of magnitude less finicky than the alternatives and can't imagine going back.
- kortex 6y ago
  I do most of my development in docker. It's tricky to start but once you get the hang of it, it's fantastic. My goal is to one day roll something like RancherOS and provision my entire desktop environment in containers. Then you can just pick up and move it to whatever machine you wish.
- amrox 6y ago
  I wish the tooling around devcontainer.json was available separately from VS Code. Most of my company are die-hand vim users. We’ve rolled our own “devcontainer” analog but it’s not nearly as seemless.
- budafish 6y ago
  Ive never quite managed to get the hang of setting up a dev container. I'ts a bit too tricky for me. I have resorted to using a throwaway dev container which mimics the local storage of packages. Hopefully someone find the below bash function useful. The only issue I have using this method is that I cant use VSCode to Debug, which I am hoping to find a nice solution for, but nothing yet. function python() { docker run \ -it --rm \ --name python_$(pwd | sed 's#/home/##g' | sed 's#/#.#g')_$(date +"%H%M%S") \ -u $(id -u):$(id -g) \ -v "$(pwd)":"$(pwd)" \ -w "$(pwd)" \ -e PROJECT_ROOT="$(pwd)" \ -e PATH="$(pwd)/vendor/bin/:${PATH}" \ -e PYTHONUSERBASE="$(pwd)"/vendor \ -e PYTHONDONTWRITEBYTECODE=1 \ -e PIP_NO_CACHE_DIR=1 \ --net=host \ -h py-docker \ python:3-slim \ /bin/bash -c '/bin/bash --rcfile <(echo "PS1=\"[$(python --version)]:: \"") -i' }
cutler 6y ago
As a Ruby and Python developer I often wonder why Ruby doesn't need a virtual env. What did Ruby get right that Python didn't?
- johnisgood 6y ago
  Or a lot of other languages...
- cozzyd 6y ago
  well, a lot of "system" utilities are implemented in Python and I'm not aware of many in Ruby (they must exist? I just checked to see what on my system depends on Ruby and it seems like the only things are a screenruler utility, some texlive packages I've never heard of and gnome-code-assistance, whatever that is). Pip installing random crap is a great way to completely break some Linux systems.
  - toyg 6y ago
    Running pip as root is a great way to self-compromise. Would you ever run npm as root...?
    - cozzyd 6y ago
      Absolutely... I usually do python setup.py bdist_rpm if I want to install some package systemwide. Not sure it's really any safer, but it sure feels safer...
- rgoulter 6y ago
  For development, the "install everything locally" that virtual env provides. This avoids conflicting, incompatible versions of packages being installed in a global or per-user location. Python isn't at all the only language to suffer this problem. e.g. "DLL hell" and its variations like Haskell's "cabal hell". With Ruby, if its packages are installed with bundler, then these packages are installed locally in the project directory.
- renewiltord 6y ago
pjmlp 6y ago
Indeed, I got introduced to Python with version 1.6, and never used virtualenv.
snicker7 6y ago
Yet another python package management tool.
mkl95 6y ago
I recommend virtualenv and virtualenvwrapper. I usually set up a new venv like this: $ mkvirtualenv -a $(pwd) new_venv $ pip install -r requirements.txt When you want to activate this env and cd to the directory where you created it, you can simply do: $ workon new_venv That's all you need to know. It just works.
- orf 6y ago
  Honestly just use poetry.
  - sethammons 6y ago
    This comment would be vastly more valuable (to me at least) if you said why. The parent’s workflow mirrors my own and doesn’t feel onerous. Being completely unfamiliar, their site says: > Poetry either uses your configured virtualenvs or creates its own to always be isolated from your system. So it is just a pretty wrapper? I’m missing the value add.
    - curryst 6y ago
      One value add is a reduced cognitive overhead. I don't have to remember where my virtualenv is; Poetry remembers for me. The "poetry run" command is extremely nice. You can do something like "poetry run pytest" and put that in a Makefile and be done. You don't have to worry about making sure you're in the virtualenv. I basically don't drop into a full shell anymore (although "poetry shell" lets you do that) because I don't like having to juggle whether I'm in a virtualenv or not. "poetry build" will build your project into tarballs that pip can install. It's not earth-shattering, but it's nice to have it one tool. The value add is a large number of QoL improvements. I don't think there are any features that other solutions can't compete on; Poetry just makes it easier to do the same things. I would highly suggest it for anything where other people are going to work on it, because of the QoL, but for personal projects I don't think it really matters.
    - guitarbill 6y ago
      the venv management is a small aspect. pyproject.toml is nicer and saner than setup.py/requirements. easy example: dev dependencies in poetry. poetry's dependency resolver is great. can upload packages with poetry, twine not required (good for CI). works in fish shell, which virtualenvwrapper doesn't. lock files for deploying exactly the same dependency versions. those are the ones off the top of my head. use poetry, it's great.
globular-toast 6y ago
This looks interesting, but personally I don't have any problems with virtualenvs. I use pipx to install "global" tools and virtualenvwrapper to make project envs. It all works fine really.
- frostming 6y ago
  As I wrote in the post, you cannot make two tools isolated in their own environment work together(work together meaning one needs to import from the other). The only way to make it is to install one tool in the global site-packages and create the other virtualenvs with --system-sitepackages and this flag is off by default.
greatgib 6y ago
In fact this guy propose to have python be able to work like npm... Looks like a ridiculous idea to me as node and npm dependencies/node_modules is completely messy and inefficient.
smitty1e 6y ago
> things get tricky when it comes to nested venvs Never had such a requirement. Environments, like the interpreter itself, seem a singleton concept. I have used a Makefile that sources different Bash environment variables kept in files in etc/ within the venv to switch between, say, a Flask and a Gunicorn startup.
- raverbashing 6y ago
  Yes you're using nested venvs you're really making it harder on yourself. Way way harder on yourself.
  - smitty1e 6y ago
    I've used a nested git repository, with the child directory in the .gitignore of the parent. I know that there is some thing about installing git checkouts with pip, but that has not been important to me yet.
    - teitoklien 6y ago
      Why not just use git submodules feature ?
- ajford 6y ago
  I honestly can't even come up with a valid reason for something like nested venvs. At that point it seems more like someone's doing something wrong than 'things get tricky'.
  - beojan 6y ago
    If you have unlimited disk space, sure. Otherwise, it would be really nice to be able to layer venvs so you don't need to have multiple copies of a package installed.
    - esyir 6y ago
      Isn't that basically docker images.
akdor1154 6y ago
Interesting idea, and props for implementing something on top of an existing PEP.
dissent 6y ago
Thanks for writing this. I've always felt that virtualenv was bit of a placebo. I still use it from time to time though because others expect it. In fact I have used it this week on MacOS, and was annoyed to find that I can't install ipdb to my user packages and still see it in a virtualenv. Seems it's either global or local. (I think the --system-site-packages defaulting to off has it backwards, but it is probably reassuring if you don't know how the module finder works). Regarding nested virtualenvs, last time I checked there was no such thing. When nested, they always linked back to the system level python, and the cloned python had loads of dependencies on the global system. I'm not aware of a valid use case for nesting virtualenvs. There is one handy feature of virtualenv that I'm not sure __packages__ solves though: console_scripts and entry_points. setuptools will automatically create a console script under /usr/local/bin or similar which hooks in your module under site-packages. virtualenv includes a ./bin directory and adds it to your path so that these console scripts still work locally. Any thoughts on this?
- BiteCode_dev 6y ago
  -m is the replacement for console scripts. Well behaved cli should define a _ _ main _ _. Py Pytest, pylint, black, pip, poetry, mypy all have one. Jupyter one is buggy.
  - dissent 6y ago
    That's a fair point actually.
- frostming 6y ago
  __pypackages__ also has a bin/ directory and you can run the console_scripts with pdm run my_bin or by absolute path: /path/to/__pypackages__/3.8/bin/my_bin
dzonga 6y ago
props to the author of this tool. based on what I read from the blog, it's already a step above poetry, which a lot of people rave about recently, but still not good enough not to not use pip. question is, are we going to be nuking the __module__ folder like we nuke node_modules every now and then ? other than I will become an early adopter of the tools. currently using miniconda for my virtualenvs
- williamdclt 6y ago
  > are we going to be nuking the __module__ folder like we nuke node_modules every now and then ? Two thoughts: 1. Part of the good aspect of the node_modules is its nukeability. It's a good thing that we can blindly nuke and `npm i` 2. With node and react (I have 5 node projects and 2 react projects that I work on regularly), I really rarely have such error that I need to nuke the node modules. `npm i` to update dependencies and restarting the server because it got stuck in a weird state is usually the solution. I nuke a node_modules maybe every 3 months, probably less. On react-native, yeah it's still very much a thing though
JoshuaRLi 6y ago
Happy to see PEP 582 gaining traction, the last attempt I saw in this space was https://github.com/cs01/pythonloc https://github.com/cs01/pythonloc.
Traubenfuchs 6y ago
Could someone ELI7 why python is struggling so hard with dependencies and dependency management when in the Java world this is a solved problem and just works with maven, which is working so damn well it last got updated in November 2019, and the biggest fight going is whether you should use gradle as client instead of maven to access the same dependency ecosystem? I had to set up Python projects for some machine learning classes in college and it was a complete mess.
- mqus 6y ago
  I have the opinion that this is because "Deployment" is not a solved issue (or clearly defined and enforced) for Python. C has shared libraries and static compilation (bundling of dependencies) Java has .jar files which probably contain all your dependencies, go and Rust statically compile to something that has minimal dependencies and python has... no clue. Some packages you have to install via your package manager (in linux), because they have to be compiled against your local libraries, some other dependencies are somewhat included... and if you then try to also develop on the same machine you can choose between your system python libraries or installing them via pip (which makes your system weirder to deploy). Having said that, I'm no expert in python but I have done some work in it and the deployment question is something that always seemed weird and unexplained to me.
  - nicoburns 6y ago
    Python should just do what JavaScript does and install all dependencies to a project-local folder.
    - StreamBright 6y ago
      Not sure if JavaScript has anything that any other language or development tool should copy.
      - wilsonrocks 6y ago
        It is a mess... But you can often get a project running an awful lot quicker than in python
cdavid 6y ago
Regarding the installation of self-contained tools, this is a problem already solved by pipx, which is very simple to use. I wished such a system was included as part of official python installers.
frostming 6y ago
Thanks for giving a look and leaving so many reactions. This project was originally built to experiment on some new PEPs on Python packaging and it turned out working well. 1. What does PEP 582 differ from virtualenv, it is just another "virutualenv" The biggest difference, as you can see, is __pypackages__ doesn't carry an interpreter with it. This way the packages remain available even the base interpreter gets removed. 2. I prefer Poetry/virtualenvwrapper/Pipenv It is fine. If you feel comfortable with the current solution you are using, stick with it. I am not persuading that PDM is a better tool than those. Any of the said tools does a great job solving its own problems. PDM, IMO, does good in following aspects: - Plugin system, people can enhance the functionalities easily with plugins - Windows support, PDM has completion support for PowerShell out of the box. Other CLI features also work as good as *nix on Windows.
- lr1970 6y ago
  How does PDM deals with binary python modules like foo.so that in turn depend on other binary libraries like mylib.so? This is what conda manager and conda environments are great for, covering the ground for all major platforms.
dbsmith83 6y ago
Curious, what about environment variables? I like the ability to put my environment variables in my activate script. It seems like if you did things the way the author does, you would need all of your env variables to be global, which could be hazardous
- 2fast4you 6y ago
  You could replace that with a tool like direnv. It’ll automatically load your variables for a project when you enter it’s directory. https://direnv.net/ https://direnv.net/
  - dbsmith83 6y ago
    Interesting, thanks!
    - mixmastamyk 6y ago
      Script, source, and systemd units are options as well of course.
mikkelam 6y ago
Man i love the ease of this lib. I've been running between docker, poetry, venvs, and pipenv and they all just SUCK. Is there a best workflow for using this inside a container?
BiteCode_dev 6y ago
My first attempt, and I get "AttributeError: module 'toml.decoder' has no attribute 'TomlPreserveCommentDecoder'" So careful, far from stable. I reported it (https://github.com/frostming/pdm/issues/247 https://github.com/frostming/pdm/issues/247) because the project is great and I want it to succeed, but I won't put it in prod any time soon. I prefer my package manager to be stable than fancy. It's also why I don't use poetry and so on in training. There is always some bug that students will encounter later because there are so many config combinations they can encounter, but recent tools have not been as battle tested as we think: the enterprise world is a wild beast.
- frostming 6y ago
  > So careful, far from stable. True, bugs can't be exposed by a few users. People have vastly different environment setup for their Python development. System python, in-venv, pyenv, asdf, homebrew, etc. So I post it here to seek for help testing. The sematic versioning may be a bit misleading, it is just because there was a big breaking change to support PEP 621, and I marked it as "Alpha" stage.
  - BiteCode_dev 6y ago
    I have a lot of respect for the way you react to this comment, and how quickly you fixed the bug. Thank you for working on such a useful project, and so seriously. I immediately encountered a second different error, and I'll also take the time to craft a good bug report for it.
blt 6y ago
I struggled with Python dependency hell often until I started using Conda. I wish it were faster, but it rarely surprises me.
- arbol 6y ago
  Conda is so slow that I resorted to using it for env creation and then just pip install everything. I don't have 10+ minutes to wait for package to be installed, only to be told it can't resolve conflicts. Package version conflict is not such a big deal, just pip install a different version of the offending package!

169 points by gilad 6y ago

> One day a minor release of Python is out It is decidedly not true that I want to update my venv for every minor version bump! I deploy to a cloud service w/ a specific version; my package manager is slow to update; I develop collaboratively using a shared container w/ a fixed version. Updating shared resources with every new release is not always realistic, and that makes it so that I do need (or at least want) to use a virtualenv.

dvfjsdhgfv 6y ago
> It is decidedly not true that I want to update my venv for every minor version bump! I see your point, but testing against every minor version is a great way to spot potential problems (that you'll need to deal with anyway).
swiftcoder 6y ago
> It is decidedly not true that I want to update my venv for every minor version bump! Until you run into a minor version bump that patches a critical security vulnerability, and then you suddenly may very much want to (and in some corporate environments, have a regulatory requirement to)
- michaericalribo 6y ago
  That’s not every minor bump, though, and is worth the extra effort! Also, my use case is 100% non critical from a security standpoint, so I can afford to be careless... your point definitely stands in sensitive environments

neolog 6y ago

Sure you can make your own package directory and develop a new package manager to point into it. But what's the point when Poetry already exists?

cutler 6y ago
As the authors said - no more virtual envs.
- danmur 6y ago
  A virtualenv is just a bunch of packages, you end up with the same thing whatever you want to call it. It is probably a more roundabout solution that it needs to be, but I don't find many problems myself (also using poetry).
  - cutler 6y ago
    He also said virtual envs install their own copy of Python. No need for that either. Coming from other languages the whole virtual env thing seems alien especially the way it messes with your shell.
    - iandinwoodie 6y ago
      A virtualenv (venv) does not install its own copy of Python. Instead, when you create the venv it puts symlinks for the calling python in the <venv>/bin directory. When you activate the venv the <venv>/bin is added to the front of your PATH so that any "python" commands use the symlink'ed version of Python corresponding to your venv.
      - aequitas 6y ago
        Indeed, a virtual env is really not that exciting or complicated. Even "activating" a virtualenv is just changing your shell's PATH so it picks the virtualenv binaries, nothing else. You can also just call those binaries directly from their installed location in the env without ever having to "activate" or "workon" anything.
    - danmur 6y ago
      Activating the venv is completely optional. I never do it, but some people seem to like it.

darkteflon 6y ago

This was a good article and looks like an interesting project. Regardless of whether or not you choose to _deploy_ with Docker, _developing_ in Docker containers using the VS Code Remote extensions really solved all of Python's (and Javascript's) annoying packaging problems for me, with the bonus that you also get to specify any additional (non-Python) dependencies right there in the repo Dockerfile and have the development environment reproducible across machines and between developers. YMMV, of course, but I find this setup an order of magnitude less finicky than the alternatives and can't imagine going back.

kortex 6y ago
I do most of my development in docker. It's tricky to start but once you get the hang of it, it's fantastic. My goal is to one day roll something like RancherOS and provision my entire desktop environment in containers. Then you can just pick up and move it to whatever machine you wish.
amrox 6y ago
I wish the tooling around devcontainer.json was available separately from VS Code. Most of my company are die-hand vim users. We’ve rolled our own “devcontainer” analog but it’s not nearly as seemless.
budafish 6y ago
Ive never quite managed to get the hang of setting up a dev container. I'ts a bit too tricky for me. I have resorted to using a throwaway dev container which mimics the local storage of packages. Hopefully someone find the below bash function useful. The only issue I have using this method is that I cant use VSCode to Debug, which I am hoping to find a nice solution for, but nothing yet. function python() { docker run \ -it --rm \ --name python_$(pwd | sed 's#/home/##g' | sed 's#/#.#g')_$(date +"%H%M%S") \ -u $(id -u):$(id -g) \ -v "$(pwd)":"$(pwd)" \ -w "$(pwd)" \ -e PROJECT_ROOT="$(pwd)" \ -e PATH="$(pwd)/vendor/bin/:${PATH}" \ -e PYTHONUSERBASE="$(pwd)"/vendor \ -e PYTHONDONTWRITEBYTECODE=1 \ -e PIP_NO_CACHE_DIR=1 \ --net=host \ -h py-docker \ python:3-slim \ /bin/bash -c '/bin/bash --rcfile <(echo "PS1=\"[$(python --version)]:: \"") -i' }

cutler 6y ago

As a Ruby and Python developer I often wonder why Ruby doesn't need a virtual env. What did Ruby get right that Python didn't?

johnisgood 6y ago
Or a lot of other languages...
cozzyd 6y ago
well, a lot of "system" utilities are implemented in Python and I'm not aware of many in Ruby (they must exist? I just checked to see what on my system depends on Ruby and it seems like the only things are a screenruler utility, some texlive packages I've never heard of and gnome-code-assistance, whatever that is). Pip installing random crap is a great way to completely break some Linux systems.
- toyg 6y ago
  Running pip as root is a great way to self-compromise. Would you ever run npm as root...?
  - cozzyd 6y ago
    Absolutely... I usually do python setup.py bdist_rpm if I want to install some package systemwide. Not sure it's really any safer, but it sure feels safer...
rgoulter 6y ago
For development, the "install everything locally" that virtual env provides. This avoids conflicting, incompatible versions of packages being installed in a global or per-user location. Python isn't at all the only language to suffer this problem. e.g. "DLL hell" and its variations like Haskell's "cabal hell". With Ruby, if its packages are installed with bundler, then these packages are installed locally in the project directory.
renewiltord 6y ago

pjmlp 6y ago

Indeed, I got introduced to Python with version 1.6, and never used virtualenv.

snicker7 6y ago

Yet another python package management tool.

mkl95 6y ago

I recommend virtualenv and virtualenvwrapper. I usually set up a new venv like this: $ mkvirtualenv -a $(pwd) new_venv $ pip install -r requirements.txt When you want to activate this env and cd to the directory where you created it, you can simply do: $ workon new_venv That's all you need to know. It just works.

orf 6y ago
Honestly just use poetry.
- sethammons 6y ago
  This comment would be vastly more valuable (to me at least) if you said why. The parent’s workflow mirrors my own and doesn’t feel onerous. Being completely unfamiliar, their site says: > Poetry either uses your configured virtualenvs or creates its own to always be isolated from your system. So it is just a pretty wrapper? I’m missing the value add.
  - curryst 6y ago
    One value add is a reduced cognitive overhead. I don't have to remember where my virtualenv is; Poetry remembers for me. The "poetry run" command is extremely nice. You can do something like "poetry run pytest" and put that in a Makefile and be done. You don't have to worry about making sure you're in the virtualenv. I basically don't drop into a full shell anymore (although "poetry shell" lets you do that) because I don't like having to juggle whether I'm in a virtualenv or not. "poetry build" will build your project into tarballs that pip can install. It's not earth-shattering, but it's nice to have it one tool. The value add is a large number of QoL improvements. I don't think there are any features that other solutions can't compete on; Poetry just makes it easier to do the same things. I would highly suggest it for anything where other people are going to work on it, because of the QoL, but for personal projects I don't think it really matters.
  - guitarbill 6y ago
    the venv management is a small aspect. pyproject.toml is nicer and saner than setup.py/requirements. easy example: dev dependencies in poetry. poetry's dependency resolver is great. can upload packages with poetry, twine not required (good for CI). works in fish shell, which virtualenvwrapper doesn't. lock files for deploying exactly the same dependency versions. those are the ones off the top of my head. use poetry, it's great.

globular-toast 6y ago

This looks interesting, but personally I don't have any problems with virtualenvs. I use pipx to install "global" tools and virtualenvwrapper to make project envs. It all works fine really.

frostming 6y ago
As I wrote in the post, you cannot make two tools isolated in their own environment work together(work together meaning one needs to import from the other). The only way to make it is to install one tool in the global site-packages and create the other virtualenvs with --system-sitepackages and this flag is off by default.

greatgib 6y ago

In fact this guy propose to have python be able to work like npm... Looks like a ridiculous idea to me as node and npm dependencies/node_modules is completely messy and inefficient.

smitty1e 6y ago

> things get tricky when it comes to nested venvs Never had such a requirement. Environments, like the interpreter itself, seem a singleton concept. I have used a Makefile that sources different Bash environment variables kept in files in etc/ within the venv to switch between, say, a Flask and a Gunicorn startup.

raverbashing 6y ago
Yes you're using nested venvs you're really making it harder on yourself. Way way harder on yourself.
- smitty1e 6y ago
  I've used a nested git repository, with the child directory in the .gitignore of the parent. I know that there is some thing about installing git checkouts with pip, but that has not been important to me yet.
  - teitoklien 6y ago
    Why not just use git submodules feature ?
ajford 6y ago
I honestly can't even come up with a valid reason for something like nested venvs. At that point it seems more like someone's doing something wrong than 'things get tricky'.
- beojan 6y ago
  If you have unlimited disk space, sure. Otherwise, it would be really nice to be able to layer venvs so you don't need to have multiple copies of a package installed.
  - esyir 6y ago
    Isn't that basically docker images.

akdor1154 6y ago

Interesting idea, and props for implementing something on top of an existing PEP.

dissent 6y ago

Thanks for writing this. I've always felt that virtualenv was bit of a placebo. I still use it from time to time though because others expect it. In fact I have used it this week on MacOS, and was annoyed to find that I can't install ipdb to my user packages and still see it in a virtualenv. Seems it's either global or local. (I think the --system-site-packages defaulting to off has it backwards, but it is probably reassuring if you don't know how the module finder works). Regarding nested virtualenvs, last time I checked there was no such thing. When nested, they always linked back to the system level python, and the cloned python had loads of dependencies on the global system. I'm not aware of a valid use case for nesting virtualenvs. There is one handy feature of virtualenv that I'm not sure __packages__ solves though: console_scripts and entry_points. setuptools will automatically create a console script under /usr/local/bin or similar which hooks in your module under site-packages. virtualenv includes a ./bin directory and adds it to your path so that these console scripts still work locally. Any thoughts on this?

BiteCode_dev 6y ago
-m is the replacement for console scripts. Well behaved cli should define a _ _ main _ _. Py Pytest, pylint, black, pip, poetry, mypy all have one. Jupyter one is buggy.
- dissent 6y ago
  That's a fair point actually.
frostming 6y ago
__pypackages__ also has a bin/ directory and you can run the console_scripts with pdm run my_bin or by absolute path: /path/to/__pypackages__/3.8/bin/my_bin

dzonga 6y ago

props to the author of this tool. based on what I read from the blog, it's already a step above poetry, which a lot of people rave about recently, but still not good enough not to not use pip. question is, are we going to be nuking the __module__ folder like we nuke node_modules every now and then ? other than I will become an early adopter of the tools. currently using miniconda for my virtualenvs

williamdclt 6y ago
> are we going to be nuking the __module__ folder like we nuke node_modules every now and then ? Two thoughts: 1. Part of the good aspect of the node_modules is its nukeability. It's a good thing that we can blindly nuke and `npm i` 2. With node and react (I have 5 node projects and 2 react projects that I work on regularly), I really rarely have such error that I need to nuke the node modules. `npm i` to update dependencies and restarting the server because it got stuck in a weird state is usually the solution. I nuke a node_modules maybe every 3 months, probably less. On react-native, yeah it's still very much a thing though

JoshuaRLi 6y ago

Happy to see PEP 582 gaining traction, the last attempt I saw in this space was https://github.com/cs01/pythonloc https://github.com/cs01/pythonloc.

Traubenfuchs 6y ago

Could someone ELI7 why python is struggling so hard with dependencies and dependency management when in the Java world this is a solved problem and just works with maven, which is working so damn well it last got updated in November 2019, and the biggest fight going is whether you should use gradle as client instead of maven to access the same dependency ecosystem? I had to set up Python projects for some machine learning classes in college and it was a complete mess.

mqus 6y ago
I have the opinion that this is because "Deployment" is not a solved issue (or clearly defined and enforced) for Python. C has shared libraries and static compilation (bundling of dependencies) Java has .jar files which probably contain all your dependencies, go and Rust statically compile to something that has minimal dependencies and python has... no clue. Some packages you have to install via your package manager (in linux), because they have to be compiled against your local libraries, some other dependencies are somewhat included... and if you then try to also develop on the same machine you can choose between your system python libraries or installing them via pip (which makes your system weirder to deploy). Having said that, I'm no expert in python but I have done some work in it and the deployment question is something that always seemed weird and unexplained to me.
- nicoburns 6y ago
  Python should just do what JavaScript does and install all dependencies to a project-local folder.
  - StreamBright 6y ago
    Not sure if JavaScript has anything that any other language or development tool should copy.
    - wilsonrocks 6y ago
      It is a mess... But you can often get a project running an awful lot quicker than in python

cdavid 6y ago

Regarding the installation of self-contained tools, this is a problem already solved by pipx, which is very simple to use. I wished such a system was included as part of official python installers.

frostming 6y ago

Thanks for giving a look and leaving so many reactions. This project was originally built to experiment on some new PEPs on Python packaging and it turned out working well. 1. What does PEP 582 differ from virtualenv, it is just another "virutualenv" The biggest difference, as you can see, is __pypackages__ doesn't carry an interpreter with it. This way the packages remain available even the base interpreter gets removed. 2. I prefer Poetry/virtualenvwrapper/Pipenv It is fine. If you feel comfortable with the current solution you are using, stick with it. I am not persuading that PDM is a better tool than those. Any of the said tools does a great job solving its own problems. PDM, IMO, does good in following aspects: - Plugin system, people can enhance the functionalities easily with plugins - Windows support, PDM has completion support for PowerShell out of the box. Other CLI features also work as good as *nix on Windows.

lr1970 6y ago
How does PDM deals with binary python modules like foo.so that in turn depend on other binary libraries like mylib.so? This is what conda manager and conda environments are great for, covering the ground for all major platforms.

dbsmith83 6y ago

Curious, what about environment variables? I like the ability to put my environment variables in my activate script. It seems like if you did things the way the author does, you would need all of your env variables to be global, which could be hazardous

2fast4you 6y ago
You could replace that with a tool like direnv. It’ll automatically load your variables for a project when you enter it’s directory. https://direnv.net/ https://direnv.net/
- dbsmith83 6y ago
  Interesting, thanks!
  - mixmastamyk 6y ago
    Script, source, and systemd units are options as well of course.

mikkelam 6y ago

Man i love the ease of this lib. I've been running between docker, poetry, venvs, and pipenv and they all just SUCK. Is there a best workflow for using this inside a container?

BiteCode_dev 6y ago

My first attempt, and I get "AttributeError: module 'toml.decoder' has no attribute 'TomlPreserveCommentDecoder'" So careful, far from stable. I reported it (https://github.com/frostming/pdm/issues/247 https://github.com/frostming/pdm/issues/247) because the project is great and I want it to succeed, but I won't put it in prod any time soon. I prefer my package manager to be stable than fancy. It's also why I don't use poetry and so on in training. There is always some bug that students will encounter later because there are so many config combinations they can encounter, but recent tools have not been as battle tested as we think: the enterprise world is a wild beast.

frostming 6y ago
> So careful, far from stable. True, bugs can't be exposed by a few users. People have vastly different environment setup for their Python development. System python, in-venv, pyenv, asdf, homebrew, etc. So I post it here to seek for help testing. The sematic versioning may be a bit misleading, it is just because there was a big breaking change to support PEP 621, and I marked it as "Alpha" stage.
- BiteCode_dev 6y ago
  I have a lot of respect for the way you react to this comment, and how quickly you fixed the bug. Thank you for working on such a useful project, and so seriously. I immediately encountered a second different error, and I'll also take the time to craft a good bug report for it.

blt 6y ago

I struggled with Python dependency hell often until I started using Conda. I wish it were faster, but it rarely surprises me.

arbol 6y ago
Conda is so slow that I resorted to using it for env creation and then just pip install everything. I don't have 10+ minutes to wait for package to be installed, only to be told it can't resolve conflicts. Package version conflict is not such a big deal, just pip install a different version of the offending package!