5 ms·

I'm assuming by system you mean OS, which is a terrible, terrible idea. Dev stack and system libs should not coexist, especially because system libs should be v

I'm assuming by system you mean OS, which is a terrible, terrible idea. Dev stack and system libs should not coexist, especially because system libs should be vetted by the OS vendor, but you can't ask them to do that for dev libs. > I have to create a new conda environment for almost every ML paper that comes out That's how it's supposed to work: one env per project. As for the rest, it's more telling about the C/C++ community building the things bellow the python wrappers.

dheera 2y ago
> one env per project That causes 50 copies of the exact same version of a 1GB library to exist on my system that are all obtained from the same authority (PyPI). I have literally 50 copies of the entire set of CUDA libraries because every conda environment installs PyTorch and PyTorch includes its own CUDA. I'm not asking the OS to maintain this, but rather the package manager ("npm" or "pip" or similar) should do so on a system-wide basis. "python" and "pip" should allow for 1 copy per officially-released version of each package to live on the system, and multiple officially-released version numbers to coexist in /usr/lib. If a dev version is being used or any version that deviates from what is on PyPI, then that should live within the project.
- forrestthewoods 2y ago
  > but rather the package manager ("npm" or "pip" or similar) should do so on a system-wide basis. I basically agree with this. With the caveat that programs should not use any system search paths and packages should be hardlinked into the project directory structure from a centralized cache. This also means that a dev version looks identical to a centralized version - both are just directories within the project.
  - bomewish 2y ago
    Are you just describing something close to Nix?? In any case, Nix solves a lot of these problems.
    - forrestthewoods 2y ago
      Kind of, but not really. Nix is extremely complicated. Programs / projects including their dependencies is exceedingly simple. Also, Windows is my primary dev environment. Any solution must work cross-platform and cross-distro. Telling everyone to use a specific distro is not a solution.
      - xyzsparetimexyz 2y ago
        Nix != NixOS. It runs on WSL: https://github.com/nix-community/NixOS-WSL https://github.com/nix-community/NixOS-WSL
        forrestthewoods 2y ago
        Less than zero interest in WSL. Nix fans are becoming as obnoxious as Rust fans. And I say that as an times annoying Rust fan.
      - bomewish 2y ago
        It is complicated... but honestly I have found claude 3.5 to just 'fix it'. So you hardly have to spend much time spelunking. You just give it all your dependencies and tell it what you want. It'll whip up a working flake in a few iterations. Kinda magic. So yeah when you can abstract out the complexity it moves the needle enough to make it worth it.
- skeledrew 2y ago
  Actually conda creates hardlinks for the packages that it manages. Found this out a few weeks ago when I tried migrating my envs to another system with an identical hierarchy and ended up with a broken mess.
- olejorgenb 2y ago
  I don't think that's true for the exact same version: https://stackoverflow.com/a/57718049 https://stackoverflow.com/a/57718049 (ie. it's deduplicated)
- BiteCode_dev 2y ago
  Ah, sorry I misunderstood. Yes, it would be nice to have that by default. In fact, it's what uv (https://github.com/astral-sh/uv https://github.com/astral-sh/uv) does, and one of the reasons it's so fast and became so popular so quickly. Astral for the win.

dheera 2y ago

> one env per project That causes 50 copies of the exact same version of a 1GB library to exist on my system that are all obtained from the same authority (PyPI). I have literally 50 copies of the entire set of CUDA libraries because every conda environment installs PyTorch and PyTorch includes its own CUDA. I'm not asking the OS to maintain this, but rather the package manager ("npm" or "pip" or similar) should do so on a system-wide basis. "python" and "pip" should allow for 1 copy per officially-released version of each package to live on the system, and multiple officially-released version numbers to coexist in /usr/lib. If a dev version is being used or any version that deviates from what is on PyPI, then that should live within the project.

forrestthewoods 2y ago
> but rather the package manager ("npm" or "pip" or similar) should do so on a system-wide basis. I basically agree with this. With the caveat that programs should not use any system search paths and packages should be hardlinked into the project directory structure from a centralized cache. This also means that a dev version looks identical to a centralized version - both are just directories within the project.
- bomewish 2y ago
  Are you just describing something close to Nix?? In any case, Nix solves a lot of these problems.
  - forrestthewoods 2y ago
    Kind of, but not really. Nix is extremely complicated. Programs / projects including their dependencies is exceedingly simple. Also, Windows is my primary dev environment. Any solution must work cross-platform and cross-distro. Telling everyone to use a specific distro is not a solution.
    - xyzsparetimexyz 2y ago
      Nix != NixOS. It runs on WSL: https://github.com/nix-community/NixOS-WSL https://github.com/nix-community/NixOS-WSL
      - forrestthewoods 2y ago
        Less than zero interest in WSL. Nix fans are becoming as obnoxious as Rust fans. And I say that as an times annoying Rust fan.
    - bomewish 2y ago
      It is complicated... but honestly I have found claude 3.5 to just 'fix it'. So you hardly have to spend much time spelunking. You just give it all your dependencies and tell it what you want. It'll whip up a working flake in a few iterations. Kinda magic. So yeah when you can abstract out the complexity it moves the needle enough to make it worth it.
skeledrew 2y ago
Actually conda creates hardlinks for the packages that it manages. Found this out a few weeks ago when I tried migrating my envs to another system with an identical hierarchy and ended up with a broken mess.
olejorgenb 2y ago
I don't think that's true for the exact same version: https://stackoverflow.com/a/57718049 https://stackoverflow.com/a/57718049 (ie. it's deduplicated)
BiteCode_dev 2y ago
Ah, sorry I misunderstood. Yes, it would be nice to have that by default. In fact, it's what uv (https://github.com/astral-sh/uv https://github.com/astral-sh/uv) does, and one of the reasons it's so fast and became so popular so quickly. Astral for the win.