9 ms·
There is very little reason for a build node to need to pull 200 images in 6 hours, and here is why: When a machine issues a ``docker build`` command, the prog
by bsamuels 6y ago
There is very little reason for a build node to need to pull 200 images in 6 hours, and here is why:
When a machine issues a ``docker build`` command, the program reads the relevant dockerfile to check for any base images that need to be pulled (a la "FROM:")
These base images are identified based on the image repository, image name, and image tag. The first thing docker does is it checks its local registry and tries to find a match for the base image the docker build is requesting. If a matching image is located in the local registry, it uses that one in lieu of downloading the image.
This is significant - if your organization only uses a few dozen base images from DockerHub, those images will only be downloaded by each build node _once_, then never again.
Many docker users erroneously believe that if their Dockerfile requests a "latest" tagged image, docker build will always download the newest version of the image.
However, the "latest" tag is literally just a tag, it doesn't have any special functionality built in. If the docker build command finds an image tagged "latest" in the local registry, it stops there.
The only way to get docker build to always use the "actual latest" version of the base image is to add the "--pull" parameter to the docker build command. This arg will tell docker build to check the repository remote to see if the SHA hash of the image tagged "latest" has changed, and if so, re-download and use it. In the absolute worst case, this means each build node will pull 1 copy of each base image when the base image is updated. So unless you use 200 different base images that all have updates deployed to Dockerhub each and every day, you are fine.
- adrianmsmith 6y ago> The first thing docker does is it checks its local registry and tries to find a match for the base image the docker build is requesting. If a matching image is located in the local registry, it uses that one in lieu of downloading the image. While I agree that this is the way it's supposed to work, I have unfortunately worked at companies with "stateless" build/CI servers that download the Docker image each build.
- umvi 6y agoWell, this policy change will force them to be more efficient, and it's a net win for everyone
- Pet_Ant 6y ago> While I agree that this is the way it's supposed to work, I have unfortunately worked at companies with "stateless" build/CI servers that download the Docker image each build. Couldn't they remain stateless but be redirected through a caching proxy? Memoization is not contrary to statelessness.
- jopsen 6y agoSure, now they have to build a proxy...
- hinkley 6y agoArtifactory is less bad than most of the tools I have to use all day.
- falcolas 6y agoArtifactory is the very definition of expensive (even at an enterprise scale) when it comes to docker images though.
- hinkley 6y agoCan you tell me more? How expensive are we talking? Working for the same sized companies for a while has apparently dulled my senses. At a certain size, the capital that matters is the political capital it takes to get a vendor agreement in place to begin with. The monthly costs of the system are something you only feel through pushback on how big the repo gets, or the rate of traffic (experiencing the latter now with a browser testing SaaS)
- eblanshey 6y agoSuper easy to run a docker cache proxy: docker run -d -p 6000:5000 \ -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \ --restart always \ --name registry registry:2 That's it. Now fetch docker images from the IP that command is running on. Taken from gitlab: https://docs.gitlab.com/runner/install/registry_and_cache_servers.html https://docs.gitlab.com/runner/install/registry_and_cache_se...
- scarface74 6y agoThis is significant - if your organization only uses a few dozen base images from DockerHub, those images will only be downloaded by each build node _once_, then never again. Unless you’re using something like AWS CodeBuild that spins up a Linux/Windows container for your build environment, executes bash commands in a yaml file, and then terminates it when it is done. Nothing is stored locally after the build is finished. I’m sure there are other similar services. Wouldn’t Azure Devops using hosted builds do basically the same thing? I haven’t used it since they changed the name from Visual Studio Team Services.
- zxcvbn4038 6y agoWhat is a solution for the scenarios you have described? Amazon has ECR but it doesn’t support signing and doesn’t function as a proxy so you would miss upstream changes unless someone pushed them. Anything self hosted that supplies that functionality?
- scarface74 6y agoPay $5.00 a month. Docker is a business that deserves to get paid if they offer something valuable.
- zxcvbn4038 6y agoWell that takes all the fun out of it. Looks like Docker itself has a Dockerhub proxy - https://docs.docker.com/registry/recipes/mirror/ https://docs.docker.com/registry/recipes/mirror/ https://hackernoon.com/mirror-cache-dockerhub-locally-for-speed-f4eebd21a5ca https://hackernoon.com/mirror-cache-dockerhub-locally-for-sp... https://stackoverflow.com/questions/32531048/docker-pull-through-cache-for-private-registry-not-working https://stackoverflow.com/questions/32531048/docker-pull-thr... https://docs.docker.com/registry/configuration/ https://docs.docker.com/registry/configuration/ https://www.google.com/amp/s/ops.tips/amp/gists/aws-s3-private-docker-registry/ https://www.google.com/amp/s/ops.tips/amp/gists/aws-s3-priva... If using Alpine, looks like docker-registry is the needed package and /usr/bin/docker-registry serve /etc/docker-registry/config.yml is the command line. Next to last link has information on the config file.
- maxmcd 6y agoI'm not sure if this is 100% true any more. I've found that when enabling the DOCKER_BUILDKIT=1 env var that docker will sometimes eagerly re-fetch stale images. I think your argument is generally still true, but was happy to see that some progress is being made on dealing with stale `latest`.
- nrmitchi 6y agoI don't disagree with what you are saying, _but_: > Docker defines pull rate limits as the number of manifest requests to Docker Hub. > For example, if you already have the image, the Docker Engine client will issue a manifest request, realize it has all of the referenced layers based on the returned manifest, and stop. ... <excluded> ... So an image pull is actually one or two manifest requests, This still implies that even if you are appropriately re-using layers on your machine, with a free plan you can only do maximum 200 builds (since docker still needs to verify it has the image) per 6 hours? This change also seems to imply that builds steps which previously did not handle/require authentication against Docker hub (it was only pulling public images, and pushing elsewhere) will now be required to auth against docker hub in order to double the number of pulls/checks/builds it is allowed?
- bsamuels 6y agoThis is an excellent point. Trying to find out if docker build --pull without an accompanying blob download will trigger the rate limiter. If it does, then this will definitely be a reason to riot. It will effectively mean that anyone who wants to do more than 200 builds every 6 hours using the "right" way will have to get a docker pro subscription.
- nrmitchi 6y agoIt sounds like it definitely does trigger the rate limiter. > There is a small tradeoff – if you pull an image you already have, this is still counted even if you don’t download the layers. I expect we're just going to see a lot more recycling of build nodes once it has "used up it's docker credits".
- heavenlyblue 6y agoAll reasonable orgs should have had their private docker repo a long time ago. Everybody else is living the pipe dream where they have externalised their risk and probably deserve the Docker treatment.
- 6y ago
- AaronFriel 6y agoCI/CD systems on AWS, Azure, GCP and others might be running on Kubernetes containers (using kaniko, podman, etc) or using Docker-in-Docker, and there isn't a widely supported or in-cloud-platform tool for sharing cached layers. And as pointed out below, even if you are intelligently caching layers, manifest requests count as a pull. As far as I know, no caching proxies exist for Docker that support limiting manifest pulls.
- stingraycharles 6y agoSurprised to see Docker-in-Docker mentioned so deeply down here. It’s an extremely valid way of doing things, and non-trivial to implement a caching layer for.
- boudin 6y agoUsually docker outside of docker is used, no? If the image is cached on the host, it would be available to any container having access to the docker daemon socket as well since it's the same daemon.
- stingraycharles 6y agoNo that’s only the case if you mount the Docker socket into the container, which is not what Docker-in-Docker is.
- pbalau 6y agoIsn't Docker-in-Docker actually using the host's Docker daemon? I am mounting the docker socket in all my Docker-in-Docker containers, thus all the build tasks running on the same host can share the caches. I guess one could have docker containers that actually run docker, but I don't see a reason to do that...
- cpuguy83 6y agoNo Docker-in-Docker would generally refer to running a new dockerd inside of a container.
- solatic 6y ago> This is significant - if your organization only uses a few dozen base images from DockerHub, those images will only be downloaded by each build node _once_, then never again. You're assuming that the set of build nodes is relatively static. Plenty of architectures set up autoscaling for the underlying nodes, that terminate servers that aren't being used and relatively soon enough (tens of minutes, hours) spin up new servers to replace them as needed. Rarely do the machine images used to spin up new servers include the base images of the containers that will be spun up to replace them. Much more often, the base machine image is a base OS image, and container images are downloaded on-the-fly as needed. Essentially, the engineering cost of making image-launching more efficient was externalized onto an external provider willing to pay the price.
- grumple 6y agoIf you’re using docker for production distribution of images, you should be paying for it. That’s exactly the behavior that creates the need for a limit.
- mvanbaak 6y agoOr don’t use docker in production ;)
- closeparen 6y agoIf you're doing this, you're in a cloud environment that's also proximate to a blob store, and can trivially host your own registry.
- outworlder 6y ago> and can trivially host your own registry That is far from trivial.
- maccard 6y agoAnd now you have an alternative option - pay $5/month
- hinkley 6y ago> This is significant - if your organization only uses a few dozen base images from DockerHub, those images will only be downloaded by each build node _once_, then never again. Only if your build nodes have unlimited storage. If the build nodes are spun up on demand or have housecleaning tasks to prevent Tragedy of the Commons disk exhaustion, this is not true. On the other hand, this is what caching proxies/registries are for.
- james-mcelwain 6y agoLots of tools built on top of Docker do imbue special meaning in latest, however.
- fnord77 6y agoIf everyone in your company (plus your CI system) is behind the same firewall/IP address, that's going to be a lot more than 200 pulls.
- peterwwillis 6y agoLuckily though, companies have cash to pay for registrations, and those that won't probably have engineers who can set up Squid proxies.