4 ms·

It would be great if some HNers on the anonymous and free tiers could go through the process and report on how throttling affected them in detail. I have to ad

It would be great if some HNers on the anonymous and free tiers could go through the process and report on how throttling affected them in detail. I have to admit that 100 pulls per 6 hours doesn't sound like enough for even a small team doing CI/CD. But in the long run, it might make a lot of organizations more lean in how they do builds, which could be a good thing.

barumi 6y ago
> It would be great if some HNers on the anonymous and free tiers could go through the process and report on how throttling affected them in detail. In my POV the first fundamental change that this caused was to cease considering docker hub as reliable infrastructure, and therefore started to look for solutions to reliably source container images. For instance, GitLab already provides its own container registry, and thus it makes sense to simply source images from them than from tertiary sources like Docker Hub. Another impact that this has is that I'm looking into the need to operate a custom container registry for each Kubernetes cluster. It wouldn't be nice if an application running on Kubernetes or Docket Swarm risked downtime just because you happened to hit one of Docker Hub's rate limits.
- redis_mlc 6y ago
  Well, the throttling seems to be on anonymous and free docker hub accounts for now. So you do have the option of paid tiers. I do recall reading about the free pulls a year ago and thinking, "Wow, that's awfully generous!" :)
  - barumi 6y ago
    > Well, the throttling seems to be on anonymous and free docker hub accounts for now. So you do have the option of paid tiers. We don't need to resort to paid services. This is all about convenience. As I've said, services like GitLab already provide container registry services, and we are also free to operate our own custom container registry services. Docker Hub is only used to source reference images such as Ubuntu, postgres, rabbitmq, traefik, etc etc etc. We can simply point to a different provider to continue accessing those. Even if these projects continue to use Docker Hub then if we stick with stable releases then we can simply point to a different container registry service, even one managed by us. The main point is that Docker Hub ceased to be a reliable provider. Thus, we plan accordingly.
- chrisandchris 6y ago
  Exactly this. It‘s so easy to setup your registry using GitLab or then Docker Registry, Portainer and Traefik. No need to rely on external providers for a small team.
Sahbak 6y ago
Unless I'm mistaken, 100 pulls per 6 hours per IP is a lot - You would/should be caching most of the images from dockerhub anyways, so subsequential builds shouldn't count against your limit.
- szszrk 6y ago
  Internal repo that does caching of images is rather obvious and a good practice, but it still doesn't mean that 100 pulls are that many. Maybe we understand something different by a "small" team, but IMHO a smaller team uses more ready public images, to save time and costs. Also even not that big project on Kubernetes that includes internal software will also have a lot of other software. Let's speculate: I had a look at some teams I know and how they work (or they would like to work soon) and it's easily 15-30 images in a CI/CD pipeline. Plus tools used to develop, but not exactly released in a build (source code management, PM, communication etc). All teams I know have 2-3 cycles developed at all times (next release, hotfix, future release after the next one). This means a lot of these tools are multiplied by 2-3 (upgrading tools as well, so multiple versions of same tool needed). If you have a few clusters that have different requirements it can also be multiplied by 2 or 3. I would not dare to think what would happen if someone used "latest" tag in their daily work, or organized a docker/kubernetes workshop for new people :)
- barumi 6y ago
  > Unless I'm mistaken, 100 pulls per 6 hours per IP is a lot It really depends on your usecase. I mean, each CICD stage can involve building a brand new image and running a couple of them, sometimes more. 100 pulls per 6 hours corresponds to a constant rate of 3 min between pulls, which would require between 15 to 20 minutes for each commit. Anyway, the primary practical implications of this sort of change is a) rate limiting suddenly becomes a requirement, b) we can no longer count on Docker Hub to provide a reliable service, thus we plan accordingly.
- robrtsql 6y ago
  At one point, Docker's announcement said that all 'docker pull' requests would count towards the rate limit, even if you had every layer cached and no layers ended up being downloaded. I may be wrong but I interpreted that to mean that even caching won't prevent you from hitting the limit.
franga2000 6y ago
If your CI is doing more than 100 pulls and you have a "small team" then you must be doing something wrong. Either you have way too many project for a small team to work on, or your CI configuration is incredibly wasteful. Just a few years back I remember even the smallest Java shop would have a local "caching/forwarding" Maven repository and the upstream repos were only considered a fallback. Has this mentality not continued into the Docker era? It seems pretty easy to set up [0] and has all kinds of advantages even without the upcoming limit. [0] https://docs.docker.com/registry/recipes/mirror/ https://docs.docker.com/registry/recipes/mirror/
- barumi 6y ago
  > If your CI is doing more than 100 A limit of 100 pulls per 6 hours averages to a pull each 3 minutes. Each stage of a CICD pipeline uses more than 1 container image, and it's customary to have CICD stages build and pull from previous stages, such as integration tests,performance tests, canary tests, etc. I would dare say that you're doing something terribly wrong if you don't need to pull images often, because that largely tells you're doing barely any tests or deployments.
  - franga2000 6y ago
    The only thing that is being limited are pulls from the Docker Hub. If you're doing integration tests for a bunch of microsevices, you might have some 20 upstream images that you need to pull. Those can stay in your cache until they are updated (which is rare and predictable, because you're pinning versions, right?). You do your building and testing and all that entirely on your "own" infrastructure and then you push once test succeds.
    - barumi 6y ago
      > The only thing that is being limited are pulls from the Docker Hub. Yes. > If you're doing integration tests for a bunch of microsevices, you might have some 20 upstream images that you need to pull. Yes. > Those can stay in your cache until they are updated (which is rare and predictable, because you're pinning versions, right?). Not necessarily. You dockerize an app prior to running non-unit tests tests, and that image needs to be stored by a container registration service. With this change, Docker Hub's ceases to be that service. > You do your building and testing and all that entirely on your "own" infrastructure and then you push once test succeds. Nope, the point is that Docker Hub ceases to be relevant as a container registry. As long as you start to do all your work outside of Docker Hub, you won't need Docker Hub to do any of the work. You just get your base images from where you store your own container images and suddenly Docker Hub stops playing any role in your workflow, thus losing al relevance.
physicsguy 6y ago
The biggest issue is that if you run the command 'docker pull X', it counts as a pull even if you already have the up-to-date image, and so it doesn't do a download.

barumi 6y ago

> It would be great if some HNers on the anonymous and free tiers could go through the process and report on how throttling affected them in detail. In my POV the first fundamental change that this caused was to cease considering docker hub as reliable infrastructure, and therefore started to look for solutions to reliably source container images. For instance, GitLab already provides its own container registry, and thus it makes sense to simply source images from them than from tertiary sources like Docker Hub. Another impact that this has is that I'm looking into the need to operate a custom container registry for each Kubernetes cluster. It wouldn't be nice if an application running on Kubernetes or Docket Swarm risked downtime just because you happened to hit one of Docker Hub's rate limits.

redis_mlc 6y ago
Well, the throttling seems to be on anonymous and free docker hub accounts for now. So you do have the option of paid tiers. I do recall reading about the free pulls a year ago and thinking, "Wow, that's awfully generous!" :)
- barumi 6y ago
  > Well, the throttling seems to be on anonymous and free docker hub accounts for now. So you do have the option of paid tiers. We don't need to resort to paid services. This is all about convenience. As I've said, services like GitLab already provide container registry services, and we are also free to operate our own custom container registry services. Docker Hub is only used to source reference images such as Ubuntu, postgres, rabbitmq, traefik, etc etc etc. We can simply point to a different provider to continue accessing those. Even if these projects continue to use Docker Hub then if we stick with stable releases then we can simply point to a different container registry service, even one managed by us. The main point is that Docker Hub ceased to be a reliable provider. Thus, we plan accordingly.
chrisandchris 6y ago
Exactly this. It‘s so easy to setup your registry using GitLab or then Docker Registry, Portainer and Traefik. No need to rely on external providers for a small team.

Sahbak 6y ago

Unless I'm mistaken, 100 pulls per 6 hours per IP is a lot - You would/should be caching most of the images from dockerhub anyways, so subsequential builds shouldn't count against your limit.

szszrk 6y ago
Internal repo that does caching of images is rather obvious and a good practice, but it still doesn't mean that 100 pulls are that many. Maybe we understand something different by a "small" team, but IMHO a smaller team uses more ready public images, to save time and costs. Also even not that big project on Kubernetes that includes internal software will also have a lot of other software. Let's speculate: I had a look at some teams I know and how they work (or they would like to work soon) and it's easily 15-30 images in a CI/CD pipeline. Plus tools used to develop, but not exactly released in a build (source code management, PM, communication etc). All teams I know have 2-3 cycles developed at all times (next release, hotfix, future release after the next one). This means a lot of these tools are multiplied by 2-3 (upgrading tools as well, so multiple versions of same tool needed). If you have a few clusters that have different requirements it can also be multiplied by 2 or 3. I would not dare to think what would happen if someone used "latest" tag in their daily work, or organized a docker/kubernetes workshop for new people :)
barumi 6y ago
> Unless I'm mistaken, 100 pulls per 6 hours per IP is a lot It really depends on your usecase. I mean, each CICD stage can involve building a brand new image and running a couple of them, sometimes more. 100 pulls per 6 hours corresponds to a constant rate of 3 min between pulls, which would require between 15 to 20 minutes for each commit. Anyway, the primary practical implications of this sort of change is a) rate limiting suddenly becomes a requirement, b) we can no longer count on Docker Hub to provide a reliable service, thus we plan accordingly.
robrtsql 6y ago
At one point, Docker's announcement said that all 'docker pull' requests would count towards the rate limit, even if you had every layer cached and no layers ended up being downloaded. I may be wrong but I interpreted that to mean that even caching won't prevent you from hitting the limit.

franga2000 6y ago

If your CI is doing more than 100 pulls and you have a "small team" then you must be doing something wrong. Either you have way too many project for a small team to work on, or your CI configuration is incredibly wasteful. Just a few years back I remember even the smallest Java shop would have a local "caching/forwarding" Maven repository and the upstream repos were only considered a fallback. Has this mentality not continued into the Docker era? It seems pretty easy to set up [0] and has all kinds of advantages even without the upcoming limit. [0] https://docs.docker.com/registry/recipes/mirror/ https://docs.docker.com/registry/recipes/mirror/

barumi 6y ago
> If your CI is doing more than 100 A limit of 100 pulls per 6 hours averages to a pull each 3 minutes. Each stage of a CICD pipeline uses more than 1 container image, and it's customary to have CICD stages build and pull from previous stages, such as integration tests,performance tests, canary tests, etc. I would dare say that you're doing something terribly wrong if you don't need to pull images often, because that largely tells you're doing barely any tests or deployments.
- franga2000 6y ago
  The only thing that is being limited are pulls from the Docker Hub. If you're doing integration tests for a bunch of microsevices, you might have some 20 upstream images that you need to pull. Those can stay in your cache until they are updated (which is rare and predictable, because you're pinning versions, right?). You do your building and testing and all that entirely on your "own" infrastructure and then you push once test succeds.
  - barumi 6y ago
    > The only thing that is being limited are pulls from the Docker Hub. Yes. > If you're doing integration tests for a bunch of microsevices, you might have some 20 upstream images that you need to pull. Yes. > Those can stay in your cache until they are updated (which is rare and predictable, because you're pinning versions, right?). Not necessarily. You dockerize an app prior to running non-unit tests tests, and that image needs to be stored by a container registration service. With this change, Docker Hub's ceases to be that service. > You do your building and testing and all that entirely on your "own" infrastructure and then you push once test succeds. Nope, the point is that Docker Hub ceases to be relevant as a container registry. As long as you start to do all your work outside of Docker Hub, you won't need Docker Hub to do any of the work. You just get your base images from where you store your own container images and suddenly Docker Hub stops playing any role in your workflow, thus losing al relevance.

physicsguy 6y ago

The biggest issue is that if you run the command 'docker pull X', it counts as a pull even if you already have the up-to-date image, and so it doesn't do a download.