6 ms·

And stop putting everything on the planet into a god damn virtual environment.

rbanffy 5y ago
Virtual environments are a good way to have different packages for different applications running on different versions of Python. My usual rule is you don't mess with the system-provided Python environments for specific applications you are working with. I would even suggest dropping support for many packages at the distro level unless they are required by other non-python packages (the same way Django and Twisted are requirements for MaaS).
- 1337shadow 5y ago
  > don't mess with the system-provided Python environments for specific applications you are working with Right, but then just use pip install --user instead of a virtualenv. Actually --user is the default now when running pip install from non-root user, so, just pip install as a user will work and not mess with the system packages, just don't do sudo pip install.
  - gjulianm 5y ago
    I've run into problems when running pip install --user. Someone installs a package in the user environment, and that works. But later you'll install a package in a shared virtualenv (because a Python program needs to run as a service or by another users or whatever) and pip doesn't install it because it's already on the user environment. However, other users don't have access to that environment and they will have an import error that you won't see from your user. Not to mention that installing every dependency in the same environment is a recipe for both disaster with version conflicts and bloat when you don't really know which packages belong to which applications.
    - 1337shadow 5y ago
      Interesting problem but I think it's more a problem in virtualenv, which don't use global site package by default, I'm surprised that it uses user site packages by default. However, installing every dependency in the same environment is also what distro package managers do, I don't see that as a recipe for disaster, it all depends how well the said packages are maintained in which case just upgrading them should fix whatever problem.
      - gjulianm 5y ago
        > However, installing every dependency in the same environment is also what distro package managers do, I don't see that as a recipe for disaster, Distro packages are done in a way where it's hard to get conflicts. I haven't found an instance where python3-X and python3-Y can't be installed because one requires python3-Z v1.2 and the other python3-Z v2.1. With Python packages, that happens way more often. Also, distro package managers keep track of everything installed, while python ones (at least pip I'm 100% sure) don't. You could install a package which upgrades another one, and that breaks another that you had installed before and pip won't say a word.
        1337shadow 5y ago
        > I haven't found an instance where python3-X and python3-Y can't be installed because one requires python3-Z v1.2 and the other python3-Z v2.1. With Python packages, that happens way more often. Exactly, because the packages were created/updated at a point in time were they were compatible: a distro ships python 3.x with packages of python modules that are compatible with python 3.x. Now if you as a user want to use a python package that works with python 3.y, you either have to wait, or install python 3.y, or use a container of python 3.y. Again, if all python package were up to date as in "working together at this point in time" then `pip upgrade` would work. > Also, distro package managers keep track of everything installed, while python ones (at least pip I'm 100% sure) don't. Are you sure you checked in the `*.dist-info` directories? There should be one per package, containing: a METADATA file, with all dependency versions, a RECORD file, with every installed file and their hash, and much more! see for yourself ;) > You could install a package which upgrades another one, and that breaks another that you had installed before and pip won't say a word. Let's agree to disagree right there: - in your point of view: this is a problem in pip - in my point of view: this is a maintenance problem in the python packages themselves, caused by the general practice of pinning
  - robertlagrant 5y ago
    > Right, but then just use pip install --user instead of a virtualenv This seems almost as bad as system python. I suppose it's fine if you only work on one thing, but as soon as you don't, your dev environment will become chaotic and lots of confusing WFM will happen. E.g. this is why npm has a separate node_modules folder for each project.
    - 1337shadow 5y ago
      > why npm has a separate node_modules folder for each project This sacrifies time and disk space and hides tech debt under the carpet, I'd rather have a solution like `pip upgrade` that would upgrade all packages and fix the environment, like `pacman -Syu`, but people would have to stop pinning versions and actually maintain their codebases and the dependencies they uses.
      - robertlagrant 5y ago
        > but people would have to stop pinning versions and actually maintain their codebases and the dependencies they uses Virtual environments is the mechanism by which you do that in a non-silly way. How do you think people cope with a dependency that has a bug introduced in the most recent version?
- salawat 5y ago
  Apt and desktop environments need to break their Python dependencies then, or stop doing daft things like colliding different 3.x run levels on the same runtime path. Go ahead and install pip3 from Python 3.6. use that to install pip3 for 3.8, and try using apt. Backup first.
  - formerly_proven 5y ago
    Your last sentence doesn't parse for me at all. How exactly do you get a pip installed on Python 3.6 to modify anything about another install of Python, let alone a different version?
    - nerdponx 5y ago
      If your distro dumps all the installed files into `/usr/lib/python3` without distinguishing among 3.8, 3.9, etc.
      - rbanffy 5y ago
        > If your distro dumps all the installed files into `/usr/lib/python3` Oh dear... What kind of monster does this?
    - salawat 5y ago
      Apt install python-pip3 You'll get pip3 from Python 3.6. apt install python3.8 You'll get system-level python3. Use pip3 in 3.6 to install the latest pip3 that requires Python 3.8, and drops that package in the dist-packages folder, which is shared between system Python runtime versions. Behold as further attempts to do something sane with APT blow up, because now you have 3.8 packages on 3.6's runtime path. I've lost weeks to this particular brand of distro daftness. I thought that surely no one would do that .. Alas...
      - rbanffy 5y ago
        "Thou shalt not mess with the system Python"
nerdponx 5y ago
Why? I want virtual environments for every other programming language!
pimterry 5y ago
What don't you like about virtual environments? As far as I can tell, they're a strict requirement, if you ever work on more than one project and want to isolate the dependencies (because you need different versions of the same package, because you want to ensure you've tracked all deps, whatever). Is it that you don't isolate dependencies between your python projects at all, or is there some other solution you prefer?
gjvc 5y ago
In an enterprise world, you can't rely on dependencies being present on the target machine, so install a .venv/ directory in your application distribution containing all the libraries (populated by pip) and a bin/venv-python wrapper script to set PYTHONPATH correctly and to call the .venv/bin/python. Then create a bin/run-app script to call bin/venv-python, with the location of your entry point module from src/python/ In summary: bin/run-app -> bin/venv-python -> .venv/bin/python -- I'm not sure how to make it simpler than this. The cost is perhaps in disk space, but I don't care about that. Or don't do the above and deploy via docker and use pip to install to the system python in the image.
nvarsj 5y ago
Python is basically unusable without venvs in my experience. Not sure what your alternative suggestion is. I hope you don't think global pip install is the right way.

Virtual environments are a good way to have different packages for different applications running on different versions of Python. My usual rule is you don't mess with the system-provided Python environments for specific applications you are working with. I would even suggest dropping support for many packages at the distro level unless they are required by other non-python packages (the same way Django and Twisted are requirements for MaaS).

1337shadow 5y ago
> don't mess with the system-provided Python environments for specific applications you are working with Right, but then just use pip install --user instead of a virtualenv. Actually --user is the default now when running pip install from non-root user, so, just pip install as a user will work and not mess with the system packages, just don't do sudo pip install.
- gjulianm 5y ago
  I've run into problems when running pip install --user. Someone installs a package in the user environment, and that works. But later you'll install a package in a shared virtualenv (because a Python program needs to run as a service or by another users or whatever) and pip doesn't install it because it's already on the user environment. However, other users don't have access to that environment and they will have an import error that you won't see from your user. Not to mention that installing every dependency in the same environment is a recipe for both disaster with version conflicts and bloat when you don't really know which packages belong to which applications.
  - 1337shadow 5y ago
    Interesting problem but I think it's more a problem in virtualenv, which don't use global site package by default, I'm surprised that it uses user site packages by default. However, installing every dependency in the same environment is also what distro package managers do, I don't see that as a recipe for disaster, it all depends how well the said packages are maintained in which case just upgrading them should fix whatever problem.
    - gjulianm 5y ago
      > However, installing every dependency in the same environment is also what distro package managers do, I don't see that as a recipe for disaster, Distro packages are done in a way where it's hard to get conflicts. I haven't found an instance where python3-X and python3-Y can't be installed because one requires python3-Z v1.2 and the other python3-Z v2.1. With Python packages, that happens way more often. Also, distro package managers keep track of everything installed, while python ones (at least pip I'm 100% sure) don't. You could install a package which upgrades another one, and that breaks another that you had installed before and pip won't say a word.
      - 1337shadow 5y ago
        > I haven't found an instance where python3-X and python3-Y can't be installed because one requires python3-Z v1.2 and the other python3-Z v2.1. With Python packages, that happens way more often. Exactly, because the packages were created/updated at a point in time were they were compatible: a distro ships python 3.x with packages of python modules that are compatible with python 3.x. Now if you as a user want to use a python package that works with python 3.y, you either have to wait, or install python 3.y, or use a container of python 3.y. Again, if all python package were up to date as in "working together at this point in time" then `pip upgrade` would work. > Also, distro package managers keep track of everything installed, while python ones (at least pip I'm 100% sure) don't. Are you sure you checked in the `*.dist-info` directories? There should be one per package, containing: a METADATA file, with all dependency versions, a RECORD file, with every installed file and their hash, and much more! see for yourself ;) > You could install a package which upgrades another one, and that breaks another that you had installed before and pip won't say a word. Let's agree to disagree right there: - in your point of view: this is a problem in pip - in my point of view: this is a maintenance problem in the python packages themselves, caused by the general practice of pinning
- robertlagrant 5y ago
  > Right, but then just use pip install --user instead of a virtualenv This seems almost as bad as system python. I suppose it's fine if you only work on one thing, but as soon as you don't, your dev environment will become chaotic and lots of confusing WFM will happen. E.g. this is why npm has a separate node_modules folder for each project.
  - 1337shadow 5y ago
    > why npm has a separate node_modules folder for each project This sacrifies time and disk space and hides tech debt under the carpet, I'd rather have a solution like `pip upgrade` that would upgrade all packages and fix the environment, like `pacman -Syu`, but people would have to stop pinning versions and actually maintain their codebases and the dependencies they uses.
    - robertlagrant 5y ago
      > but people would have to stop pinning versions and actually maintain their codebases and the dependencies they uses Virtual environments is the mechanism by which you do that in a non-silly way. How do you think people cope with a dependency that has a bug introduced in the most recent version?
salawat 5y ago
Apt and desktop environments need to break their Python dependencies then, or stop doing daft things like colliding different 3.x run levels on the same runtime path. Go ahead and install pip3 from Python 3.6. use that to install pip3 for 3.8, and try using apt. Backup first.
- formerly_proven 5y ago
  Your last sentence doesn't parse for me at all. How exactly do you get a pip installed on Python 3.6 to modify anything about another install of Python, let alone a different version?
  - nerdponx 5y ago
    If your distro dumps all the installed files into `/usr/lib/python3` without distinguishing among 3.8, 3.9, etc.
    - rbanffy 5y ago
      > If your distro dumps all the installed files into `/usr/lib/python3` Oh dear... What kind of monster does this?
  - salawat 5y ago
    Apt install python-pip3 You'll get pip3 from Python 3.6. apt install python3.8 You'll get system-level python3. Use pip3 in 3.6 to install the latest pip3 that requires Python 3.8, and drops that package in the dist-packages folder, which is shared between system Python runtime versions. Behold as further attempts to do something sane with APT blow up, because now you have 3.8 packages on 3.6's runtime path. I've lost weeks to this particular brand of distro daftness. I thought that surely no one would do that .. Alas...
    - rbanffy 5y ago
      "Thou shalt not mess with the system Python"

nerdponx 5y ago

Why? I want virtual environments for every other programming language!

pimterry 5y ago

What don't you like about virtual environments? As far as I can tell, they're a strict requirement, if you ever work on more than one project and want to isolate the dependencies (because you need different versions of the same package, because you want to ensure you've tracked all deps, whatever). Is it that you don't isolate dependencies between your python projects at all, or is there some other solution you prefer?

gjvc 5y ago

In an enterprise world, you can't rely on dependencies being present on the target machine, so install a .venv/ directory in your application distribution containing all the libraries (populated by pip) and a bin/venv-python wrapper script to set PYTHONPATH correctly and to call the .venv/bin/python. Then create a bin/run-app script to call bin/venv-python, with the location of your entry point module from src/python/ In summary: bin/run-app -> bin/venv-python -> .venv/bin/python -- I'm not sure how to make it simpler than this. The cost is perhaps in disk space, but I don't care about that. Or don't do the above and deploy via docker and use pip to install to the system python in the image.

nvarsj 5y ago

Python is basically unusable without venvs in my experience. Not sure what your alternative suggestion is. I hope you don't think global pip install is the right way.