24 ms·

I run tens of thousands of docker images in production, or rather, tens of thousands of copies of a few hundred images. If you do something like this, you abso

I run tens of thousands of docker images in production, or rather, tens of thousands of copies of a few hundred images. If you do something like this, you absolutely MUST have a local registry. Harbor [1], JFrog [2], and Quay [3] would be the first ones that I look at. Harbor is open source, free, and a member of the CNCF. You will need to do a little bit of work to set it up to scale properly. JFrog offers a SaaS registry, but you will pay big $$ based on pull traffic. Their commercial site license is about $3k/year. Quay is older than either of them, stable, and high quality. I'd start with Harbor these days. [1] https://goharbor.io/ https://goharbor.io/ [2] https://www.jfrog.com/confluence/display/JFROG/JFrog+Artifactory https://www.jfrog.com/confluence/display/JFROG/JFrog+Artifac... [3] https://quay.io/ https://quay.io/

drzaiusx11 6y ago
I was a happy user of JFrog's registries via site license at my last 2 places. Seemed to just work as expected. Didn't have visibility into the cost though (other teams set it up) so I had no idea it was $3k/year.
manquer 6y ago
Just to add all the major cloud service providers provide registries ACR /ECR/GCR etc . If you run k8s service with one of the them in my experience it is best to use the corresponding registry. I have pulled and run 20k times a 1GB image in less than 10-15 minutes without breaking a sweat. Finally GitHub packages offers a registry out of the box . It is great for CI and devs to access . I generally have the tags mirrored from tags GitHub for production to ACR .
- ztjio 6y ago
  Looking at doing GitHub Packages for direct-to-dev and mirroring into ECR over here. Seems sound. But also considering other options as ECR is a pain to work with. That said, word of warning for anyone looking at GitHub Packages for docker registry: it's broken with containerd and some other similar tools. They (GitHub) are currently working on a fix: https://github.com/containerd/containerd/issues/3291 https://github.com/containerd/containerd/issues/3291
  - lolinder 6y ago
    I got set up with ECR without any difficulties whatsoever. You do have to authenticate before pulls and pushes, but that can be scripted very easily.
- threeseed 6y ago
  Github Docker Registry is a mess and should be avoided at all costs. 1) It is broken and unusable on Kubernetes and Docker Swarm. 2) It is flaky often returning 500 type errors. 3) It is expensive as the amount of pull bandwidth is very limited.
  - manquer 6y ago
    Github packages works with Github CI out of the box, it makes development lot easier, like I mentioned for best networking in prod you should always use the registry from your k8s Provider, mirroring the Github registry to ECR/GCR/ACR is fairly straightforward. Bandwidth costs are eliminated, network is lot more reliable intra DC.
  - neurostimulant 6y ago
    > It is broken and unusable on Kubernetes and Docker Swarm. Hmm, I use them on several kubernetes clusters in the past few months and don't see any issue yet.
- hamiltont 6y ago
  FYI, Using ECR with Docker Swarm is something we did try. It was hellish. We never nailed down the exact problems, but we spent about a month with 2-3 experienced engineers trying to fix the edge case issues. The main issue was ECR has a slightly different authentication model than docker swarm. The whole '--with-registry-auth' only partially works when you are using ECR. Unfortunately, it works just enough that you think it's working, until all your tokens time out and a worker can suddenly no longer pull an image. Our common failure case was an image becoming unhealthy or a node being drained. When that image would try to be restarted on a different worker, if that worker did not have the image it would try to get it from the registry. If the tokens were expired it would fail. The only "fix" we ever found was to setup a cron job that forcibly deployed a new version of a "replicated globally" image every X minutes (where X was based on ECR token expiration). It kind of worked, but we still had occasional failures we could not identify. I wish it worked better, because it was nice to use ECR. Frankly token expiration sounds much more secure too, but without direct support for token refresh inside the docker engine it's just hard to get everything to work
- deleted 6y ago
  [deleted]
apple4ever 6y ago
We have not had good luck with Quay. They are not stable, especially as of late. There was a period last month where for two weeks pulling images was a crapshoot.
hamiltont 6y ago
Thank you very much. This is exactly the type of info I needed.

drzaiusx11 6y ago

I was a happy user of JFrog's registries via site license at my last 2 places. Seemed to just work as expected. Didn't have visibility into the cost though (other teams set it up) so I had no idea it was $3k/year.

manquer 6y ago

Just to add all the major cloud service providers provide registries ACR /ECR/GCR etc . If you run k8s service with one of the them in my experience it is best to use the corresponding registry. I have pulled and run 20k times a 1GB image in less than 10-15 minutes without breaking a sweat. Finally GitHub packages offers a registry out of the box . It is great for CI and devs to access . I generally have the tags mirrored from tags GitHub for production to ACR .

ztjio 6y ago
Looking at doing GitHub Packages for direct-to-dev and mirroring into ECR over here. Seems sound. But also considering other options as ECR is a pain to work with. That said, word of warning for anyone looking at GitHub Packages for docker registry: it's broken with containerd and some other similar tools. They (GitHub) are currently working on a fix: https://github.com/containerd/containerd/issues/3291 https://github.com/containerd/containerd/issues/3291
- lolinder 6y ago
  I got set up with ECR without any difficulties whatsoever. You do have to authenticate before pulls and pushes, but that can be scripted very easily.
threeseed 6y ago
Github Docker Registry is a mess and should be avoided at all costs. 1) It is broken and unusable on Kubernetes and Docker Swarm. 2) It is flaky often returning 500 type errors. 3) It is expensive as the amount of pull bandwidth is very limited.
- manquer 6y ago
  Github packages works with Github CI out of the box, it makes development lot easier, like I mentioned for best networking in prod you should always use the registry from your k8s Provider, mirroring the Github registry to ECR/GCR/ACR is fairly straightforward. Bandwidth costs are eliminated, network is lot more reliable intra DC.
- neurostimulant 6y ago
  > It is broken and unusable on Kubernetes and Docker Swarm. Hmm, I use them on several kubernetes clusters in the past few months and don't see any issue yet.
hamiltont 6y ago
FYI, Using ECR with Docker Swarm is something we did try. It was hellish. We never nailed down the exact problems, but we spent about a month with 2-3 experienced engineers trying to fix the edge case issues. The main issue was ECR has a slightly different authentication model than docker swarm. The whole '--with-registry-auth' only partially works when you are using ECR. Unfortunately, it works just enough that you think it's working, until all your tokens time out and a worker can suddenly no longer pull an image. Our common failure case was an image becoming unhealthy or a node being drained. When that image would try to be restarted on a different worker, if that worker did not have the image it would try to get it from the registry. If the tokens were expired it would fail. The only "fix" we ever found was to setup a cron job that forcibly deployed a new version of a "replicated globally" image every X minutes (where X was based on ECR token expiration). It kind of worked, but we still had occasional failures we could not identify. I wish it worked better, because it was nice to use ECR. Frankly token expiration sounds much more secure too, but without direct support for token refresh inside the docker engine it's just hard to get everything to work
deleted 6y ago
[deleted]

apple4ever 6y ago

We have not had good luck with Quay. They are not stable, especially as of late. There was a period last month where for two weeks pulling images was a crapshoot.

hamiltont 6y ago

Thank you very much. This is exactly the type of info I needed.