3 ms·
This sucks for individuals and open source. For folks that have a heavy reliance on dockerhub, here are some things that may help (not all are applicable to all
by suryao 2y ago
This sucks for individuals and open source. For folks that have a heavy reliance on dockerhub, here are some things that may help (not all are applicable to all use cases):
1. Setup a pull through mirror. Google Artifact Registry has decent limits and good coverage for public images. This requires just one config change and can be very useful to mitigate rate limits if you're using popular images cached in GAR.[1]
2. Setup a private pull through image registry for private images. This will require renaming all the images in your build and deployment scripts and can get very cumbersome.
3. Get your IPs allowlisted by Docker, especially if you can't have docker auth on the servers. The pricing for this can be very high. Rough numbers: $20,000/year for 5 IPs and usually go upwards of $50k/year.
4. Setup a transparent docker hub mirror. This is great because no changes need to be made to pipelines except one minor config change (similar to 1). We wrote a blog about how this can be done using the official docker registry image and AWS.[2] It is very important to NOT use the official docker registry image [3] as that itself can get throttled and lead to hairy issues. Host your own fork of the registry image and use that instead.
We spent a lot of time researching this for certain use cases while building infrastructure for serving Github actions at WarpBuild.
Hope this helps.
[1] https://cloud.google.com/artifact-registry/docs/pull-cached-dockerhub-images https://cloud.google.com/artifact-registry/docs/pull-cached-...
[2] https://www.warpbuild.com/blog/docker-mirror-setup https://www.warpbuild.com/blog/docker-mirror-setup
[3] https://hub.docker.com/_/registry https://hub.docker.com/_/registry
- knallfrosch 2y agoRegister for free and you get a higher limit: 40 pulls is plenty. What do you imagine running that requires more than 40 dockerhub (not local) pulls on an hourly basis?
- tekno45 2y agoif i start an eks cluster in a NAT environment with 10 nodes and 4 daemon sets. I need 40 pulls by default. Lots of tutorials out there to do this that will no longer work as well.
- merb 2y agodoes eks pull k8s stuff from docker.io? I thought k8s images are not on docker.io, I doubt that aws puts their special images there.
- tekno45 2y agoi really meant any managed k8s by default anything you need from helm charts will be pulled from docker hub. and its normal to have a storage daemon, networking agents, loggers on every node so if you launch enough at once during an autoscale event, you'd trigger this limit.
- merb 2y agoyeah but you can easily create a mirror for that and maybe use an admission controller that mutates it and/or validates all images
- Shish2k 2y agoMy modest homelab is currently running 42 unique images, and it seems "checking for updates" counts as a pull even if it doesn't download anything, and the hourly limits will kick in even if I only run `docker compose pull` once a month...