5 ms·

Bear in mind this isn't just end-users installing on their machines, it also includes continuous integration scripts that run quite frequently.

westurner 5y ago
[Huge GPU] packages can be cached locally: persist ~/.cache/pip between builds with e.g. Docker, run a PyPI caching proxy, "[Discussions on Python.org] [Packaging] Draft PEP: PyPI cost solutions: CI, mirrors, containers, and caching to scale" https://discuss.python.org/t/draft-pep-pypi-cost-solutions-ci-mirrors-containers-and-caching-to-scale/3681 https://discuss.python.org/t/draft-pep-pypi-cost-solutions-c... > Continuous Integration automated build and testing services can help reduce the costs of hosting PyPI by running local mirrors and advising clients in regards to how to efficiently re-build software hundreds or thousands of times a month without re-downloading everything from PyPI every time. [...] > Request from and advisory for CI Services and CI Implementors: > Dear CI Service, > - Please consider running local package mirrors and enabling use of local package mirrors by default for clients’ CI builds. > - Please advise clients regarding more efficient containerized software build and test strategies. > Running local package mirrors will save PyPI (the Python Package Index, a service maintained by PyPA, a group within the non-profit Python Software Foundation) generously donated resources. (At present (March 2020), PyPI costs ~ $800,000 USD a month to operate; even with generously donated resources). Looks like the current figure is significantly higher than $800K/mo for science. How to persist ~/.cache/pip between builds with e.g. Docker in order to minimize unnecessary GPU package re-downloads: RUN --mount=type=cache,target=/root/.cache/pip RUN --mount=type=cache,target=/home/appuser/.cache/pip
- remram 5y ago
  From an open-source developer's perspective, whether we hit PyPI (a third-party free service) or the cache provided by the CI service (a third-party free service) doesn't seem very different.
  - captn3m0 5y ago
    A 7TB local cache within the CI service is much cheaper to host, since it doesn't have CDN concerns, and bandwidth is much cheaper "within the infra".
- kortex 5y ago
  Wow, that RUN trick is exactly what I've been looking for! I've spent hours and hours in Docker documentation and hadn't seen that functionality. Looks like it might be buildkit-specific?
  - westurner 5y ago
    AFAIU `RUN --mount=type=cache` is specific to moby (Docker Engine) with `DOCKER_BUILDKIT=1`, though buildah does support a build-time volume mount option and moby easily could as well: "build time only -v option" https://github.com/moby/moby/issues/14080 https://github.com/moby/moby/issues/14080 "Build images with BuildKit" https://docs.docker.com/develop/develop-images/build_enhancements/ https://docs.docker.com/develop/develop-images/build_enhance...
viraptor 5y ago
> includes continuous integration scripts that run quite frequently This use case is something I believe they could charge for if they need to cover infra costs. Same as Docker hub started doing - if someone fails to cache properly in their CI and wants to redownload things from the internet, they should pay for that.

westurner 5y ago

[Huge GPU] packages can be cached locally: persist ~/.cache/pip between builds with e.g. Docker, run a PyPI caching proxy, "[Discussions on Python.org] [Packaging] Draft PEP: PyPI cost solutions: CI, mirrors, containers, and caching to scale" https://discuss.python.org/t/draft-pep-pypi-cost-solutions-ci-mirrors-containers-and-caching-to-scale/3681 https://discuss.python.org/t/draft-pep-pypi-cost-solutions-c... > Continuous Integration automated build and testing services can help reduce the costs of hosting PyPI by running local mirrors and advising clients in regards to how to efficiently re-build software hundreds or thousands of times a month without re-downloading everything from PyPI every time. [...] > Request from and advisory for CI Services and CI Implementors: > Dear CI Service, > - Please consider running local package mirrors and enabling use of local package mirrors by default for clients’ CI builds. > - Please advise clients regarding more efficient containerized software build and test strategies. > Running local package mirrors will save PyPI (the Python Package Index, a service maintained by PyPA, a group within the non-profit Python Software Foundation) generously donated resources. (At present (March 2020), PyPI costs ~ $800,000 USD a month to operate; even with generously donated resources). Looks like the current figure is significantly higher than $800K/mo for science. How to persist ~/.cache/pip between builds with e.g. Docker in order to minimize unnecessary GPU package re-downloads: RUN --mount=type=cache,target=/root/.cache/pip RUN --mount=type=cache,target=/home/appuser/.cache/pip

remram 5y ago
From an open-source developer's perspective, whether we hit PyPI (a third-party free service) or the cache provided by the CI service (a third-party free service) doesn't seem very different.
- captn3m0 5y ago
  A 7TB local cache within the CI service is much cheaper to host, since it doesn't have CDN concerns, and bandwidth is much cheaper "within the infra".
kortex 5y ago
Wow, that RUN trick is exactly what I've been looking for! I've spent hours and hours in Docker documentation and hadn't seen that functionality. Looks like it might be buildkit-specific?
- westurner 5y ago
  AFAIU `RUN --mount=type=cache` is specific to moby (Docker Engine) with `DOCKER_BUILDKIT=1`, though buildah does support a build-time volume mount option and moby easily could as well: "build time only -v option" https://github.com/moby/moby/issues/14080 https://github.com/moby/moby/issues/14080 "Build images with BuildKit" https://docs.docker.com/develop/develop-images/build_enhancements/ https://docs.docker.com/develop/develop-images/build_enhance...

viraptor 5y ago

> includes continuous integration scripts that run quite frequently This use case is something I believe they could charge for if they need to cover infra costs. Same as Docker hub started doing - if someone fails to cache properly in their CI and wants to redownload things from the internet, they should pay for that.