4 ms·

I must be missing something because I understood that pip cached packages already. Perhaps it still creates a copy of the package files in the virtual environm

I must be missing something because I understood that pip cached packages already. Perhaps it still creates a copy of the package files in the virtual environment, thus the cache only saves repeated downloads and not local disk space. If that's the case then this does look really useful.

zahlman 2y ago
Pip caches downloads, but in the sort of filesystem database that git uses. You can't even directly pull .whl files (nor unzipped folders) out of it. Every virtual environment gets its own, unzipped copy of the wheels represented in that database. So yes, it's only "saving repeated downloads". (And it will still talk to PyPI by default to see if there's a newer version. There's an outstanding issue to add a proper "offline mode": https://github.com/pypa/pip/issues/8057 https://github.com/pypa/pip/issues/8057) It doesn't need to work like that. A lot of libraries would work fine directly from the wheel, because they're essentially renamed zip files and Python knows how to import from the archive contents directly. But this doesn't work if the package is supposed to come with any mutable data, nor if it tries to use ordinary file I/O for its immutable data (you're supposed to use a standard library helper for that, but awareness is poor and it's a hassle anyway). Long ago, the "egg" format expected you to set a "zip-safe" flag (via Setuptools, back when it was your actual packaging tool rather than just a behind-the-scenes helper wrapped in multiple backwards-compatibility layers) so that installers could choose to leave the archive zipped. But I don't know that Pip ever actually used that information, and it was easy to get wrong. But more importantly, the contents of the virtual environment could be referenced from a cache that contained actual wheels (packed or unpacked) by hard links, `.pth` files (with a slight performance hit) or symlinks (I'm pretty sure; haven't tested).

zahlman 2y ago

Pip caches downloads, but in the sort of filesystem database that git uses. You can't even directly pull .whl files (nor unzipped folders) out of it. Every virtual environment gets its own, unzipped copy of the wheels represented in that database. So yes, it's only "saving repeated downloads". (And it will still talk to PyPI by default to see if there's a newer version. There's an outstanding issue to add a proper "offline mode": https://github.com/pypa/pip/issues/8057 https://github.com/pypa/pip/issues/8057) It doesn't need to work like that. A lot of libraries would work fine directly from the wheel, because they're essentially renamed zip files and Python knows how to import from the archive contents directly. But this doesn't work if the package is supposed to come with any mutable data, nor if it tries to use ordinary file I/O for its immutable data (you're supposed to use a standard library helper for that, but awareness is poor and it's a hassle anyway). Long ago, the "egg" format expected you to set a "zip-safe" flag (via Setuptools, back when it was your actual packaging tool rather than just a behind-the-scenes helper wrapped in multiple backwards-compatibility layers) so that installers could choose to leave the archive zipped. But I don't know that Pip ever actually used that information, and it was easy to get wrong. But more importantly, the contents of the virtual environment could be referenced from a cache that contained actual wheels (packed or unpacked) by hard links, `.pth` files (with a slight performance hit) or symlinks (I'm pretty sure; haven't tested).