4 ms·
I wish pip had some package deduplication implemented. Even some basic local environments have >100MB of dependencies. ML environments go into the gigabytes ran
by bno1 4y ago
I wish pip had some package deduplication implemented. Even some basic local environments have >100MB of dependencies. ML environments go into the gigabytes range from what I remember.
- smeagull 4y agoCargo allows you to share a single package directory. Having a single site-packages and the ability to import a particular version (or latest by default) would solve this.
- Asraelite 4y agoThis is also nice because you can clean up your build files all in one go. Every now and again I delete my cargo target directory and get 5GB back.
- BerislavLopac 4y agoI'm not sure what you mean - unlike npm, pip installs only one copy of a dependency in each environment.
- duckmysick 4y agoSo if I have environments A, B, and C and each has a dependency ML-package-1.2 that's 100 MB, it means there's one copy of it in each environment? Meaning 3x100 MB?
- dagw 4y agoMeaning 3x100 MB? By default, yes. It is possible to install ML-package-1.2 into your 'base' python and then have virtual environments A,B and C all use that instead of installing their own copies. However this is generally not considered best practice.
- BerislavLopac 4y agoAh, so across multiple environments? You do have a point, but that has nothing to do with pip - it's not an environment manager. I can see that as an option in tools like venv or poetry, possibly setting a "common" location to share across environments.