3 ms·
[Huge GPU] packages can be cached locally: persist ~/.cache/pip between builds with e.g. Docker, run a PyPI caching proxy, "[Discussions on Python.org] [Packag
by westurner 5y ago
[Huge GPU] packages can be cached locally: persist ~/.cache/pip between builds with e.g. Docker, run a PyPI caching proxy,
"[Discussions on Python.org] [Packaging] Draft PEP: PyPI cost solutions: CI, mirrors, containers, and caching to scale"
https://discuss.python.org/t/draft-pep-pypi-cost-solutions-ci-mirrors-containers-and-caching-to-scale/3681 https://discuss.python.org/t/draft-pep-pypi-cost-solutions-c...
> Continuous Integration automated build and testing services
can help reduce the costs of hosting PyPI by running local mirrors
and advising clients in regards to how to efficiently re-build
software hundreds or thousands of times a month without re-downloading everything from PyPI every time.
[...]
> Request from and advisory for CI Services and CI Implementors:
> Dear CI Service,
> - Please consider running local package mirrors and enabling use of local package mirrors by default for clients’ CI builds.
> - Please advise clients regarding more efficient containerized
software build and test strategies.
> Running local package mirrors will save PyPI (the Python Package Index, a service maintained by PyPA, a group within the non-profit Python
Software Foundation) generously donated resources. (At present (March 2020), PyPI costs ~ $800,000 USD a month to operate; even with generously donated resources).
Looks like the current figure is significantly higher than $800K/mo for science.
How to persist ~/.cache/pip between builds with e.g. Docker in order to minimize unnecessary GPU package re-downloads:
RUN --mount=type=cache,target=/root/.cache/pip
RUN --mount=type=cache,target=/home/appuser/.cache/pip
- remram 5y agoFrom an open-source developer's perspective, whether we hit PyPI (a third-party free service) or the cache provided by the CI service (a third-party free service) doesn't seem very different.
- captn3m0 5y agoA 7TB local cache within the CI service is much cheaper to host, since it doesn't have CDN concerns, and bandwidth is much cheaper "within the infra".
- kortex 5y agoWow, that RUN trick is exactly what I've been looking for! I've spent hours and hours in Docker documentation and hadn't seen that functionality. Looks like it might be buildkit-specific?
- westurner 5y agoAFAIU `RUN --mount=type=cache` is specific to moby (Docker Engine) with `DOCKER_BUILDKIT=1`, though buildah does support a build-time volume mount option and moby easily could as well: "build time only -v option" https://github.com/moby/moby/issues/14080 https://github.com/moby/moby/issues/14080 "Build images with BuildKit" https://docs.docker.com/develop/develop-images/build_enhancements/ https://docs.docker.com/develop/develop-images/build_enhance...