4 ms·

If your CI is doing more than 100 pulls and you have a "small team" then you must be doing something wrong. Either you have way too many project for a small tea

by franga2000 6y ago

If your CI is doing more than 100 pulls and you have a "small team" then you must be doing something wrong. Either you have way too many project for a small team to work on, or your CI configuration is incredibly wasteful. Just a few years back I remember even the smallest Java shop would have a local "caching/forwarding" Maven repository and the upstream repos were only considered a fallback. Has this mentality not continued into the Docker era? It seems pretty easy to set up [0] and has all kinds of advantages even without the upcoming limit. [0] https://docs.docker.com/registry/recipes/mirror/ https://docs.docker.com/registry/recipes/mirror/

barumi 6y ago
> If your CI is doing more than 100 A limit of 100 pulls per 6 hours averages to a pull each 3 minutes. Each stage of a CICD pipeline uses more than 1 container image, and it's customary to have CICD stages build and pull from previous stages, such as integration tests,performance tests, canary tests, etc. I would dare say that you're doing something terribly wrong if you don't need to pull images often, because that largely tells you're doing barely any tests or deployments.
- franga2000 6y ago
  The only thing that is being limited are pulls from the Docker Hub. If you're doing integration tests for a bunch of microsevices, you might have some 20 upstream images that you need to pull. Those can stay in your cache until they are updated (which is rare and predictable, because you're pinning versions, right?). You do your building and testing and all that entirely on your "own" infrastructure and then you push once test succeds.
  - barumi 6y ago
    > The only thing that is being limited are pulls from the Docker Hub. Yes. > If you're doing integration tests for a bunch of microsevices, you might have some 20 upstream images that you need to pull. Yes. > Those can stay in your cache until they are updated (which is rare and predictable, because you're pinning versions, right?). Not necessarily. You dockerize an app prior to running non-unit tests tests, and that image needs to be stored by a container registration service. With this change, Docker Hub's ceases to be that service. > You do your building and testing and all that entirely on your "own" infrastructure and then you push once test succeds. Nope, the point is that Docker Hub ceases to be relevant as a container registry. As long as you start to do all your work outside of Docker Hub, you won't need Docker Hub to do any of the work. You just get your base images from where you store your own container images and suddenly Docker Hub stops playing any role in your workflow, thus losing al relevance.
    - franga2000 6y ago
      It looks like you don't, but I've always seen Docker Hub as...well...a hub, not just any ol' registry. It's where people publish final builds of the images that they release to the world and others pull them from there. If you're using it for basically passing your own images between CI workers or whatever other workload that requires you to constantly be pushing and pulling, you should be paying them for it or hosting your own registry. If you're in a situation where having your own registry means Docker Hub "loses all relevance", then you're probably not the kind of user they have any reason to care about (and that's nonsense anyway, because you'll still almost certainly be using 3rd party images from there).
      - barumi 6y ago
        > It looks like you don't, I'm sorry to say but your insistence of speaking out of a position of blisful ignorance and your refusal to open your eyes does not change the world around you. > but I've always seen Docker Hub as...well...a hub, not just any ol' registry. No one with the faintest cursory knowledge of Docker would ever claim that Docker Hub is anything other than a docker registry. Docker itself advertises Docker Hub as a service "to create, manage, and deliver your teams' container applications." Docker itself describes Docker Hub as a "cloud-based application registry and development team collaboration services." That's their own official marketing slogan, taken straight out of their "Why Docker" page. And even that you managed to miss and failed to understand. > It's where people publish final builds It really isn't. That's not how containers work at all, and that's not how Docker promotes the service, and that's not how anyone uses it. Nothing in your comments is even remotely compatible with basic notions of CICD. Apparently you're so out of touch with containerization and CICD and devops workflow that you're even entirely ignorant of basic concepts like what a latest tag is supposed to represent and how tags are used to update apps. The fact that you try to pass off that absurd assertion in such absolute terms makes it quite clear that you have absolutely no idea about what you're talking about. Hell, your claim can only possibly be made by someone who never even registered a user account in Docker Hub. Please refrain from speaking about stuff you know nothing about. > If you're using it for basically passing your own images between CI workers or whatever other workload that requires you to constantly be pushing and pulling This mindless statement just makes it clear as water that you have no idea of how containers are used. At all. And yet here you are degrading the signal/noise ratio by blabbering about stuff you know nothing about.
- benmmurphy 6y ago
  we hit this because we are using AWS codebuild and the public ips are presumably reused between different builds.
jacques_chester 6y ago
> If your CI is doing more than 100 pulls and you have a "small team" then you must be doing something wrong. Most CI systems poll to see if there are new versions of an image. If this is done with a GET request (to get the manifest as a first step towards pulling layers), it counts towards the rate limiting. > Has this mentality not continued into the Docker era? It seems pretty easy to set up [0] and has all kinds of advantages even without the upcoming limit. Maven separates the domain of a Maven repository from the coordinates of packages. Docker does not. Dockerhub's domain is inferred into any shorthand reference like `foo/bar`, meaning that it will ignore a pull-through cache. You can ostensibly configure a docker daemon to treat another registry as a trusted mirror, but it's harder than it looks in practice, because on most production systems that won't (and shouldn't) be enabled.
- franga2000 6y ago
  Ok, if just checking whether a newer version is available counts towards the rate limit, than that is just completely insane! EDIT: Yeah, that makes sense given the protocol's design. Anyways, HEAD requests don't count, so hopefully the client can utilise those (or be modified to). [/EDIT] And yes, Maven has a design much better suited to having a local "proxy" (in fact, it always does that on localhost anyways), but the setup looks pretty straightforward and I really don't see much reason to not use it on all dev, test and CI machines. I'd also use it in production, but even if you'd prefer not to, production servers don't pull that often and updates are usually done through hooks, not constantly polling a repo. So basically, from my perspective (handful of servers, each with some 20 containers) this change will be thoroughly annoying, but will force me to set up caching which will make things much faster and also decrease the reliance on an external service for day-to-day operation. Not a fan of it, but yi completely see the reason why they did it.
  - jacques_chester 6y ago
    My point is that the name of the image and the location of the registry are mixed together. If it was easy to do cache hierarchies folks wouldn't be so fussed.

barumi 6y ago

> If your CI is doing more than 100 A limit of 100 pulls per 6 hours averages to a pull each 3 minutes. Each stage of a CICD pipeline uses more than 1 container image, and it's customary to have CICD stages build and pull from previous stages, such as integration tests,performance tests, canary tests, etc. I would dare say that you're doing something terribly wrong if you don't need to pull images often, because that largely tells you're doing barely any tests or deployments.

franga2000 6y ago
The only thing that is being limited are pulls from the Docker Hub. If you're doing integration tests for a bunch of microsevices, you might have some 20 upstream images that you need to pull. Those can stay in your cache until they are updated (which is rare and predictable, because you're pinning versions, right?). You do your building and testing and all that entirely on your "own" infrastructure and then you push once test succeds.
- barumi 6y ago
  > The only thing that is being limited are pulls from the Docker Hub. Yes. > If you're doing integration tests for a bunch of microsevices, you might have some 20 upstream images that you need to pull. Yes. > Those can stay in your cache until they are updated (which is rare and predictable, because you're pinning versions, right?). Not necessarily. You dockerize an app prior to running non-unit tests tests, and that image needs to be stored by a container registration service. With this change, Docker Hub's ceases to be that service. > You do your building and testing and all that entirely on your "own" infrastructure and then you push once test succeds. Nope, the point is that Docker Hub ceases to be relevant as a container registry. As long as you start to do all your work outside of Docker Hub, you won't need Docker Hub to do any of the work. You just get your base images from where you store your own container images and suddenly Docker Hub stops playing any role in your workflow, thus losing al relevance.
  - franga2000 6y ago
    It looks like you don't, but I've always seen Docker Hub as...well...a hub, not just any ol' registry. It's where people publish final builds of the images that they release to the world and others pull them from there. If you're using it for basically passing your own images between CI workers or whatever other workload that requires you to constantly be pushing and pulling, you should be paying them for it or hosting your own registry. If you're in a situation where having your own registry means Docker Hub "loses all relevance", then you're probably not the kind of user they have any reason to care about (and that's nonsense anyway, because you'll still almost certainly be using 3rd party images from there).
    - barumi 6y ago
      > It looks like you don't, I'm sorry to say but your insistence of speaking out of a position of blisful ignorance and your refusal to open your eyes does not change the world around you. > but I've always seen Docker Hub as...well...a hub, not just any ol' registry. No one with the faintest cursory knowledge of Docker would ever claim that Docker Hub is anything other than a docker registry. Docker itself advertises Docker Hub as a service "to create, manage, and deliver your teams' container applications." Docker itself describes Docker Hub as a "cloud-based application registry and development team collaboration services." That's their own official marketing slogan, taken straight out of their "Why Docker" page. And even that you managed to miss and failed to understand. > It's where people publish final builds It really isn't. That's not how containers work at all, and that's not how Docker promotes the service, and that's not how anyone uses it. Nothing in your comments is even remotely compatible with basic notions of CICD. Apparently you're so out of touch with containerization and CICD and devops workflow that you're even entirely ignorant of basic concepts like what a latest tag is supposed to represent and how tags are used to update apps. The fact that you try to pass off that absurd assertion in such absolute terms makes it quite clear that you have absolutely no idea about what you're talking about. Hell, your claim can only possibly be made by someone who never even registered a user account in Docker Hub. Please refrain from speaking about stuff you know nothing about. > If you're using it for basically passing your own images between CI workers or whatever other workload that requires you to constantly be pushing and pulling This mindless statement just makes it clear as water that you have no idea of how containers are used. At all. And yet here you are degrading the signal/noise ratio by blabbering about stuff you know nothing about.
benmmurphy 6y ago
we hit this because we are using AWS codebuild and the public ips are presumably reused between different builds.

jacques_chester 6y ago

> If your CI is doing more than 100 pulls and you have a "small team" then you must be doing something wrong. Most CI systems poll to see if there are new versions of an image. If this is done with a GET request (to get the manifest as a first step towards pulling layers), it counts towards the rate limiting. > Has this mentality not continued into the Docker era? It seems pretty easy to set up [0] and has all kinds of advantages even without the upcoming limit. Maven separates the domain of a Maven repository from the coordinates of packages. Docker does not. Dockerhub's domain is inferred into any shorthand reference like `foo/bar`, meaning that it will ignore a pull-through cache. You can ostensibly configure a docker daemon to treat another registry as a trusted mirror, but it's harder than it looks in practice, because on most production systems that won't (and shouldn't) be enabled.

franga2000 6y ago
Ok, if just checking whether a newer version is available counts towards the rate limit, than that is just completely insane! EDIT: Yeah, that makes sense given the protocol's design. Anyways, HEAD requests don't count, so hopefully the client can utilise those (or be modified to). [/EDIT] And yes, Maven has a design much better suited to having a local "proxy" (in fact, it always does that on localhost anyways), but the setup looks pretty straightforward and I really don't see much reason to not use it on all dev, test and CI machines. I'd also use it in production, but even if you'd prefer not to, production servers don't pull that often and updates are usually done through hooks, not constantly polling a repo. So basically, from my perspective (handful of servers, each with some 20 containers) this change will be thoroughly annoying, but will force me to set up caching which will make things much faster and also decrease the reliance on an external service for day-to-day operation. Not a fan of it, but yi completely see the reason why they did it.
- jacques_chester 6y ago
  My point is that the name of the image and the location of the registry are mixed together. If it was easy to do cache hierarchies folks wouldn't be so fussed.