4 ms·

This is part of why I install my Python dependencies from downstream Linux distro repos. I never use virtualenv. If a distro is missing a package I need, it's a

This is part of why I install my Python dependencies from downstream Linux distro repos. I never use virtualenv. If a distro is missing a package I need, it's a simple process to put it together, and the additional steps and checks built into the process stop close to 100% of these issues. Getting a human here also lets you do things like patch out telemetry or other anti-features. Software repositories without a human review process are a bloody stupid idea.

cozzyd 7y ago
Indeed. Plus this whole "install the whole python ecosystem for each thing you want to use" is insane.
- gdevenyi 7y ago
  In science this is pretty important so we can control versions and reproduce a result again as well.
  - MetalGuru 7y ago
    There are other, more efficient ways of handling dependency version conflicts than having an isolated env where each module is downloaded specifically for that env. For example, it doesn't make sense that if I have two virtual env's that ,use the exact same module (and version), it's downloaded and stored twice on my machine.
    - hipnopotam 7y ago
      That's one of the issues that conda (https://docs.conda.io/en/latest/ https://docs.conda.io/en/latest/) solves by design. When using conda environments you get hard links whenever possible. Improvements to venv, making it built-in module, and projects designed to simplify dependencies made conda less attractive in comparison but it's still a solid way to have Python. Still, it doesn't really solve fundamental issue with Python packages you mentioned, unfortunately.
  - cozzyd 7y ago
    For real reproducibility you want to go a step further with a virtual machine or something. Freezing your python dependencies won't shield you from changes in your C standard library or differences in vector extensions or whatever. In practice, most codes shouldn't be so fragile though if dependencies are reasonably well behaved... Otherwise how can you even trust that the dependency is giving a reasonable answer?
- pfranz 7y ago
  I'm a bit torn and often err on the side of installing into --user or virtualenv. What often happened is I would toy around with a project, install a few libraries, then abandon it. Months/years later I would hit an incompatibility and I wouldn't know which packages were necessary for things I use everyday and which were because of that abandoned package. Similarly, when revisiting that abandoned project I wouldn't know what a good known set of libraries were. Installing stuff globally also makes it really difficult to know what dependencies were needed to run it on a different machine. This wasn't just for Python libraries, but also with OS packages (especially non-sanctioned RPMs). I've been very selective about what I install globally, and usually first reach for a VM or other virtual environment. One big counter to this is when I want to make it a simple commandline tool to run. I have to enter a Virtualenv just to run my fancier `ls` command?
  - ddevault 7y ago
    My solution for this problem is apk's virtual packages. I'll do something like this: apk add -t .myproject-deps py3-foobar py3-foobaz This creates a fake ".myproject-deps" package which depends on py3-foobar and py3-foobaz, then I can just uninstall ".myproject-deps" later and it nukes the rest. `head /etc/apk/world` is generally sufficient to get a list of projects I've forgotten about whenever I feel like some spring cleaning.
Carpetsmoker 7y ago
> Software repositories without a human review process are a bloody stupid idea. What review process? Do people actually review/audit the code? Usually they don't. All it tells you that at least one person though it looked "okay enough" to package it, based on unclear criteria. It's most certainly not a "review process".
- scbrg 7y ago
  Which, honestly, would have been quite sufficient in this case. I can't remember ever having seen anything like this in Debian, for instance.

cozzyd 7y ago

Indeed. Plus this whole "install the whole python ecosystem for each thing you want to use" is insane.

gdevenyi 7y ago
In science this is pretty important so we can control versions and reproduce a result again as well.
- MetalGuru 7y ago
  There are other, more efficient ways of handling dependency version conflicts than having an isolated env where each module is downloaded specifically for that env. For example, it doesn't make sense that if I have two virtual env's that ,use the exact same module (and version), it's downloaded and stored twice on my machine.
  - hipnopotam 7y ago
    That's one of the issues that conda (https://docs.conda.io/en/latest/ https://docs.conda.io/en/latest/) solves by design. When using conda environments you get hard links whenever possible. Improvements to venv, making it built-in module, and projects designed to simplify dependencies made conda less attractive in comparison but it's still a solid way to have Python. Still, it doesn't really solve fundamental issue with Python packages you mentioned, unfortunately.
- cozzyd 7y ago
  For real reproducibility you want to go a step further with a virtual machine or something. Freezing your python dependencies won't shield you from changes in your C standard library or differences in vector extensions or whatever. In practice, most codes shouldn't be so fragile though if dependencies are reasonably well behaved... Otherwise how can you even trust that the dependency is giving a reasonable answer?
pfranz 7y ago
I'm a bit torn and often err on the side of installing into --user or virtualenv. What often happened is I would toy around with a project, install a few libraries, then abandon it. Months/years later I would hit an incompatibility and I wouldn't know which packages were necessary for things I use everyday and which were because of that abandoned package. Similarly, when revisiting that abandoned project I wouldn't know what a good known set of libraries were. Installing stuff globally also makes it really difficult to know what dependencies were needed to run it on a different machine. This wasn't just for Python libraries, but also with OS packages (especially non-sanctioned RPMs). I've been very selective about what I install globally, and usually first reach for a VM or other virtual environment. One big counter to this is when I want to make it a simple commandline tool to run. I have to enter a Virtualenv just to run my fancier `ls` command?
- ddevault 7y ago
  My solution for this problem is apk's virtual packages. I'll do something like this: apk add -t .myproject-deps py3-foobar py3-foobaz This creates a fake ".myproject-deps" package which depends on py3-foobar and py3-foobaz, then I can just uninstall ".myproject-deps" later and it nukes the rest. `head /etc/apk/world` is generally sufficient to get a list of projects I've forgotten about whenever I feel like some spring cleaning.

Carpetsmoker 7y ago

> Software repositories without a human review process are a bloody stupid idea. What review process? Do people actually review/audit the code? Usually they don't. All it tells you that at least one person though it looked "okay enough" to package it, based on unclear criteria. It's most certainly not a "review process".

scbrg 7y ago
Which, honestly, would have been quite sufficient in this case. I can't remember ever having seen anything like this in Debian, for instance.