3 ms·

Just to add all the major cloud service providers provide registries ACR /ECR/GCR etc . If you run k8s service with one of the them in my experience it is best

Just to add all the major cloud service providers provide registries ACR /ECR/GCR etc . If you run k8s service with one of the them in my experience it is best to use the corresponding registry. I have pulled and run 20k times a 1GB image in less than 10-15 minutes without breaking a sweat. Finally GitHub packages offers a registry out of the box . It is great for CI and devs to access . I generally have the tags mirrored from tags GitHub for production to ACR .

ztjio 6y ago
Looking at doing GitHub Packages for direct-to-dev and mirroring into ECR over here. Seems sound. But also considering other options as ECR is a pain to work with. That said, word of warning for anyone looking at GitHub Packages for docker registry: it's broken with containerd and some other similar tools. They (GitHub) are currently working on a fix: https://github.com/containerd/containerd/issues/3291 https://github.com/containerd/containerd/issues/3291
- lolinder 6y ago
  I got set up with ECR without any difficulties whatsoever. You do have to authenticate before pulls and pushes, but that can be scripted very easily.
threeseed 6y ago
Github Docker Registry is a mess and should be avoided at all costs. 1) It is broken and unusable on Kubernetes and Docker Swarm. 2) It is flaky often returning 500 type errors. 3) It is expensive as the amount of pull bandwidth is very limited.
- manquer 6y ago
  Github packages works with Github CI out of the box, it makes development lot easier, like I mentioned for best networking in prod you should always use the registry from your k8s Provider, mirroring the Github registry to ECR/GCR/ACR is fairly straightforward. Bandwidth costs are eliminated, network is lot more reliable intra DC.
- neurostimulant 6y ago
  > It is broken and unusable on Kubernetes and Docker Swarm. Hmm, I use them on several kubernetes clusters in the past few months and don't see any issue yet.
hamiltont 6y ago
FYI, Using ECR with Docker Swarm is something we did try. It was hellish. We never nailed down the exact problems, but we spent about a month with 2-3 experienced engineers trying to fix the edge case issues. The main issue was ECR has a slightly different authentication model than docker swarm. The whole '--with-registry-auth' only partially works when you are using ECR. Unfortunately, it works just enough that you think it's working, until all your tokens time out and a worker can suddenly no longer pull an image. Our common failure case was an image becoming unhealthy or a node being drained. When that image would try to be restarted on a different worker, if that worker did not have the image it would try to get it from the registry. If the tokens were expired it would fail. The only "fix" we ever found was to setup a cron job that forcibly deployed a new version of a "replicated globally" image every X minutes (where X was based on ECR token expiration). It kind of worked, but we still had occasional failures we could not identify. I wish it worked better, because it was nice to use ECR. Frankly token expiration sounds much more secure too, but without direct support for token refresh inside the docker engine it's just hard to get everything to work
deleted 6y ago
[deleted]

ztjio 6y ago

Looking at doing GitHub Packages for direct-to-dev and mirroring into ECR over here. Seems sound. But also considering other options as ECR is a pain to work with. That said, word of warning for anyone looking at GitHub Packages for docker registry: it's broken with containerd and some other similar tools. They (GitHub) are currently working on a fix: https://github.com/containerd/containerd/issues/3291 https://github.com/containerd/containerd/issues/3291

lolinder 6y ago
I got set up with ECR without any difficulties whatsoever. You do have to authenticate before pulls and pushes, but that can be scripted very easily.

threeseed 6y ago

Github Docker Registry is a mess and should be avoided at all costs. 1) It is broken and unusable on Kubernetes and Docker Swarm. 2) It is flaky often returning 500 type errors. 3) It is expensive as the amount of pull bandwidth is very limited.

manquer 6y ago
Github packages works with Github CI out of the box, it makes development lot easier, like I mentioned for best networking in prod you should always use the registry from your k8s Provider, mirroring the Github registry to ECR/GCR/ACR is fairly straightforward. Bandwidth costs are eliminated, network is lot more reliable intra DC.
neurostimulant 6y ago
> It is broken and unusable on Kubernetes and Docker Swarm. Hmm, I use them on several kubernetes clusters in the past few months and don't see any issue yet.

hamiltont 6y ago

FYI, Using ECR with Docker Swarm is something we did try. It was hellish. We never nailed down the exact problems, but we spent about a month with 2-3 experienced engineers trying to fix the edge case issues. The main issue was ECR has a slightly different authentication model than docker swarm. The whole '--with-registry-auth' only partially works when you are using ECR. Unfortunately, it works just enough that you think it's working, until all your tokens time out and a worker can suddenly no longer pull an image. Our common failure case was an image becoming unhealthy or a node being drained. When that image would try to be restarted on a different worker, if that worker did not have the image it would try to get it from the registry. If the tokens were expired it would fail. The only "fix" we ever found was to setup a cron job that forcibly deployed a new version of a "replicated globally" image every X minutes (where X was based on ECR token expiration). It kind of worked, but we still had occasional failures we could not identify. I wish it worked better, because it was nice to use ECR. Frankly token expiration sounds much more secure too, but without direct support for token refresh inside the docker engine it's just hard to get everything to work

deleted 6y ago

[deleted]