3 ms·

Heard of virtualenv?

cozzyd 6y ago
Yes... that's exactly what I was referring to. The problem is this: https://github.com/pypa/packaging-problems/issues/328 https://github.com/pypa/packaging-problems/issues/328
- rbanffy 6y ago
  I've been thinking of offering a patch to `fdupes` to turn duplicates into CoW copies. ZFS and BtrFS have them covered, but I'm not sure about APFS and I bet NTFS doesn't. In any case, it's really hard to get a workstation with less than a terabyte of storage in it. I have 2.9 gigs under mine and that's for 53 environments. I'm not sure that's a huge issue.
  - cozzyd 6y ago
    Sure in some use cases it doesn't matter. In others (e.g. on an HPC cluster where you have a quota in your home directory measured in GB) it does. CoW won't help on GPFS or Lustre, probably. Arguably Python has no place in HPC but a lot of people use it. And on my laptop technically I have the space but I am loathe to give it up to duplicate various packages many times.
    - aprdm 6y ago
      What would you suggest instead for HPC?
      - cozzyd 6y ago
        Well, generally you want something with better performance characteristics, which increasingly is not the case (see e.g. https://www.nature.com/articles/s41550-020-1208-y%C2%A0 https://www.nature.com/articles/s41550-020-1208-y%C2%A0). Though, even if your stuff is implemented in Fortran or C or C++ or on a GPU, Python is still a natural scripting language that is often the primary interface to some other piece of code, thanks to swig or boost::python or cppyy. But Python's fast rate of change and lack of backwards compatibility makes it problematic for this use. Our HPC cluster recently forced everyone to upgrade to Python 3 and that caused all sorts of problems for everyone. I understand there isn't really a choice with the lack of Python 2.7 support, but this has not been a great experience for anybody.
        aprdm 6y ago
        Yeah totally. Our use cases seem very similar. We still haven't transitioned to python3 at our HPC. Well.. we do have it available but only around ~10% of the workloads make use of python3 and their libraries. We have created the two different paths in our NFS and if you're in python3 then you can't access python2 and vice-versa. We will be migrating everything to python3 eventually but as you noted it is a big effort with a lot of possibility for chaos.
    - rbanffy 6y ago
      It's somewhat dangerous, but you can always replace dupes with links that work in your filesystem. Lustre, IIRC, supports hard links. I'm not sure whether GPFS does that, but I don't see any reason for that. If that fails, symlinks should work. The Python environment directories shouldn't be writable anyway.
      - cozzyd 6y ago
        > The Python environment directories shouldn't be writable anyway. Yeah, except people start putting together their own Python environments in their home directories because that's what third-party packages suggest and asking admins to install random packages globally for random one-off tasks is not really sustainable. I'm sure there are workarounds, but it would be nice if virtualenv had some automatic way of figuring this out.
        rbanffy 6y ago
        Sorry. Not what I meant. Your programs shouldn't write to these directories while running anyway, so once deduplicated, they should remain that way.
        cozzyd 6y ago
        Ah sorry! I see what you mean now. I guess it wouldn't be so hard to write a script to deduplicate a bunch of virtualenv environments, but I'm scared to be the one to go down that rabbit hole.
        rbanffy 6y ago
        There is a version of fdupes that makes hard links. Maybe it's enough for what you need.

Yes... that's exactly what I was referring to. The problem is this: https://github.com/pypa/packaging-problems/issues/328 https://github.com/pypa/packaging-problems/issues/328

rbanffy 6y ago
I've been thinking of offering a patch to `fdupes` to turn duplicates into CoW copies. ZFS and BtrFS have them covered, but I'm not sure about APFS and I bet NTFS doesn't. In any case, it's really hard to get a workstation with less than a terabyte of storage in it. I have 2.9 gigs under mine and that's for 53 environments. I'm not sure that's a huge issue.
- cozzyd 6y ago
  Sure in some use cases it doesn't matter. In others (e.g. on an HPC cluster where you have a quota in your home directory measured in GB) it does. CoW won't help on GPFS or Lustre, probably. Arguably Python has no place in HPC but a lot of people use it. And on my laptop technically I have the space but I am loathe to give it up to duplicate various packages many times.
  - aprdm 6y ago
    What would you suggest instead for HPC?
    - cozzyd 6y ago
      Well, generally you want something with better performance characteristics, which increasingly is not the case (see e.g. https://www.nature.com/articles/s41550-020-1208-y%C2%A0 https://www.nature.com/articles/s41550-020-1208-y%C2%A0). Though, even if your stuff is implemented in Fortran or C or C++ or on a GPU, Python is still a natural scripting language that is often the primary interface to some other piece of code, thanks to swig or boost::python or cppyy. But Python's fast rate of change and lack of backwards compatibility makes it problematic for this use. Our HPC cluster recently forced everyone to upgrade to Python 3 and that caused all sorts of problems for everyone. I understand there isn't really a choice with the lack of Python 2.7 support, but this has not been a great experience for anybody.
      - aprdm 6y ago
        Yeah totally. Our use cases seem very similar. We still haven't transitioned to python3 at our HPC. Well.. we do have it available but only around ~10% of the workloads make use of python3 and their libraries. We have created the two different paths in our NFS and if you're in python3 then you can't access python2 and vice-versa. We will be migrating everything to python3 eventually but as you noted it is a big effort with a lot of possibility for chaos.
  - rbanffy 6y ago
    It's somewhat dangerous, but you can always replace dupes with links that work in your filesystem. Lustre, IIRC, supports hard links. I'm not sure whether GPFS does that, but I don't see any reason for that. If that fails, symlinks should work. The Python environment directories shouldn't be writable anyway.
    - cozzyd 6y ago
      > The Python environment directories shouldn't be writable anyway. Yeah, except people start putting together their own Python environments in their home directories because that's what third-party packages suggest and asking admins to install random packages globally for random one-off tasks is not really sustainable. I'm sure there are workarounds, but it would be nice if virtualenv had some automatic way of figuring this out.
      - rbanffy 6y ago
        Sorry. Not what I meant. Your programs shouldn't write to these directories while running anyway, so once deduplicated, they should remain that way.
        cozzyd 6y ago
        Ah sorry! I see what you mean now. I guess it wouldn't be so hard to write a script to deduplicate a bunch of virtualenv environments, but I'm scared to be the one to go down that rabbit hole.
        rbanffy 6y ago
        There is a version of fdupes that makes hard links. Maybe it's enough for what you need.