3 ms·

Indeed. Plus this whole "install the whole python ecosystem for each thing you want to use" is insane.

by cozzyd 7y ago

gdevenyi 7y ago
In science this is pretty important so we can control versions and reproduce a result again as well.
- MetalGuru 7y ago
  There are other, more efficient ways of handling dependency version conflicts than having an isolated env where each module is downloaded specifically for that env. For example, it doesn't make sense that if I have two virtual env's that ,use the exact same module (and version), it's downloaded and stored twice on my machine.
  - hipnopotam 7y ago
    That's one of the issues that conda (https://docs.conda.io/en/latest/ https://docs.conda.io/en/latest/) solves by design. When using conda environments you get hard links whenever possible. Improvements to venv, making it built-in module, and projects designed to simplify dependencies made conda less attractive in comparison but it's still a solid way to have Python. Still, it doesn't really solve fundamental issue with Python packages you mentioned, unfortunately.
- cozzyd 7y ago
  For real reproducibility you want to go a step further with a virtual machine or something. Freezing your python dependencies won't shield you from changes in your C standard library or differences in vector extensions or whatever. In practice, most codes shouldn't be so fragile though if dependencies are reasonably well behaved... Otherwise how can you even trust that the dependency is giving a reasonable answer?
pfranz 7y ago
I'm a bit torn and often err on the side of installing into --user or virtualenv. What often happened is I would toy around with a project, install a few libraries, then abandon it. Months/years later I would hit an incompatibility and I wouldn't know which packages were necessary for things I use everyday and which were because of that abandoned package. Similarly, when revisiting that abandoned project I wouldn't know what a good known set of libraries were. Installing stuff globally also makes it really difficult to know what dependencies were needed to run it on a different machine. This wasn't just for Python libraries, but also with OS packages (especially non-sanctioned RPMs). I've been very selective about what I install globally, and usually first reach for a VM or other virtual environment. One big counter to this is when I want to make it a simple commandline tool to run. I have to enter a Virtualenv just to run my fancier `ls` command?
- ddevault 7y ago
  My solution for this problem is apk's virtual packages. I'll do something like this: apk add -t .myproject-deps py3-foobar py3-foobaz This creates a fake ".myproject-deps" package which depends on py3-foobar and py3-foobaz, then I can just uninstall ".myproject-deps" later and it nukes the rest. `head /etc/apk/world` is generally sufficient to get a list of projects I've forgotten about whenever I feel like some spring cleaning.

Indeed. Plus this whole "install the whole python ecosystem for each thing you want to use" is insane.

by cozzyd 7y ago

Indeed. Plus this whole "install the whole python ecosystem for each thing you want to use" is insane.

gdevenyi 7y ago

In science this is pretty important so we can control versions and reproduce a result again as well.

MetalGuru 7y ago
There are other, more efficient ways of handling dependency version conflicts than having an isolated env where each module is downloaded specifically for that env. For example, it doesn't make sense that if I have two virtual env's that ,use the exact same module (and version), it's downloaded and stored twice on my machine.
- hipnopotam 7y ago
  That's one of the issues that conda (https://docs.conda.io/en/latest/ https://docs.conda.io/en/latest/) solves by design. When using conda environments you get hard links whenever possible. Improvements to venv, making it built-in module, and projects designed to simplify dependencies made conda less attractive in comparison but it's still a solid way to have Python. Still, it doesn't really solve fundamental issue with Python packages you mentioned, unfortunately.
cozzyd 7y ago
For real reproducibility you want to go a step further with a virtual machine or something. Freezing your python dependencies won't shield you from changes in your C standard library or differences in vector extensions or whatever. In practice, most codes shouldn't be so fragile though if dependencies are reasonably well behaved... Otherwise how can you even trust that the dependency is giving a reasonable answer?

pfranz 7y ago

I'm a bit torn and often err on the side of installing into --user or virtualenv. What often happened is I would toy around with a project, install a few libraries, then abandon it. Months/years later I would hit an incompatibility and I wouldn't know which packages were necessary for things I use everyday and which were because of that abandoned package. Similarly, when revisiting that abandoned project I wouldn't know what a good known set of libraries were. Installing stuff globally also makes it really difficult to know what dependencies were needed to run it on a different machine. This wasn't just for Python libraries, but also with OS packages (especially non-sanctioned RPMs). I've been very selective about what I install globally, and usually first reach for a VM or other virtual environment. One big counter to this is when I want to make it a simple commandline tool to run. I have to enter a Virtualenv just to run my fancier `ls` command?

ddevault 7y ago
My solution for this problem is apk's virtual packages. I'll do something like this: apk add -t .myproject-deps py3-foobar py3-foobaz This creates a fake ".myproject-deps" package which depends on py3-foobar and py3-foobaz, then I can just uninstall ".myproject-deps" later and it nukes the rest. `head /etc/apk/world` is generally sufficient to get a list of projects I've forgotten about whenever I feel like some spring cleaning.