10 ms·

Do you run a local registry? Any high-quality articles/youtube talks to share? I'm about to set one up for our own little cluster (~5 machines, ~75 containers).

Do you run a local registry? Any high-quality articles/youtube talks to share? I'm about to set one up for our own little cluster (~5 machines, ~75 containers). I know tons about docker engine, and a fair bit about the registry, but it's always nice to watch a "lessons learned from actually doing this in production" talk to know what mistakes to avoid

sytse 6y ago
If you want to run a local registry to stay below the 100 pulls per 6 hours limit please consider GitLab. The Dependency Proxy https://docs.gitlab.com/ee/user/packages/dependency_proxy/ https://docs.gitlab.com/ee/user/packages/dependency_proxy/ will cache docker images. This way you stay within the limits Docker set and subsequent pulls should be faster as well.
oppositelock 6y ago
I run tens of thousands of docker images in production, or rather, tens of thousands of copies of a few hundred images. If you do something like this, you absolutely MUST have a local registry. Harbor [1], JFrog [2], and Quay [3] would be the first ones that I look at. Harbor is open source, free, and a member of the CNCF. You will need to do a little bit of work to set it up to scale properly. JFrog offers a SaaS registry, but you will pay big $$ based on pull traffic. Their commercial site license is about $3k/year. Quay is older than either of them, stable, and high quality. I'd start with Harbor these days. [1] https://goharbor.io/ https://goharbor.io/ [2] https://www.jfrog.com/confluence/display/JFROG/JFrog+Artifactory https://www.jfrog.com/confluence/display/JFROG/JFrog+Artifac... [3] https://quay.io/ https://quay.io/
- drzaiusx11 6y ago
  I was a happy user of JFrog's registries via site license at my last 2 places. Seemed to just work as expected. Didn't have visibility into the cost though (other teams set it up) so I had no idea it was $3k/year.
- manquer 6y ago
  Just to add all the major cloud service providers provide registries ACR /ECR/GCR etc . If you run k8s service with one of the them in my experience it is best to use the corresponding registry. I have pulled and run 20k times a 1GB image in less than 10-15 minutes without breaking a sweat. Finally GitHub packages offers a registry out of the box . It is great for CI and devs to access . I generally have the tags mirrored from tags GitHub for production to ACR .
  - ztjio 6y ago
    Looking at doing GitHub Packages for direct-to-dev and mirroring into ECR over here. Seems sound. But also considering other options as ECR is a pain to work with. That said, word of warning for anyone looking at GitHub Packages for docker registry: it's broken with containerd and some other similar tools. They (GitHub) are currently working on a fix: https://github.com/containerd/containerd/issues/3291 https://github.com/containerd/containerd/issues/3291
    - lolinder 6y ago
      I got set up with ECR without any difficulties whatsoever. You do have to authenticate before pulls and pushes, but that can be scripted very easily.
  - threeseed 6y ago
    Github Docker Registry is a mess and should be avoided at all costs. 1) It is broken and unusable on Kubernetes and Docker Swarm. 2) It is flaky often returning 500 type errors. 3) It is expensive as the amount of pull bandwidth is very limited.
    - manquer 6y ago
      Github packages works with Github CI out of the box, it makes development lot easier, like I mentioned for best networking in prod you should always use the registry from your k8s Provider, mirroring the Github registry to ECR/GCR/ACR is fairly straightforward. Bandwidth costs are eliminated, network is lot more reliable intra DC.
- apple4ever 6y ago
  We have not had good luck with Quay. They are not stable, especially as of late. There was a period last month where for two weeks pulling images was a crapshoot.
- hamiltont 6y ago
  Thank you very much. This is exactly the type of info I needed.
mschuster91 6y ago
Just drop a Sonatype Nexus instance on a Docker container somewhere on your network. Alternatively, use Squid if you don't push to the public Docker registry, although you might need to mess around with internal CA for SSL...
- mvanbaak 6y ago
  Nexus in a container... because storage in containers is such a good idea? Any vps with a disk is probably a better idea
  - 1337shadow 6y ago
    You can still bind mount a directory into a container...
  - Spivak 6y ago
    Storage in containers has been a long solved issue. The defaults are unfortunate because but make sense for ease of use. Your container root should be read only, ephemeral storage lives in a tmpfs or dynamic volumes depending on performance and size needs, and persistent storage lives in volumes.
- efrecon 6y ago
  I would stay away from Nexus. It has problems with latest tags.
- Spivak 6y ago
  Docker supports proxies (they call them “pullthrough repos”) so you don’t have to be so generic as an http proxy.
drzaiusx11 6y ago
I just use ECR[1] which in many cases costs less and is fully locked down behind my AWS VPC With ECR you pay for image storage: $0.09 per GB after the first 1 GB which is free [1] https://aws.amazon.com/ecr/ https://aws.amazon.com/ecr/
- tuananh 6y ago
  are you gonna rebuild all the images that you use and push to ECR?
  - drzaiusx11 6y ago
    nope. you don't have to rebuild images to push to different registries pull from docker hub once, push to ECR. then pull from ECR as much as wish
efreak 6y ago
Personally, I wish generic caching proxies were still a thing, and easier to set up. I've tried setting up squid several times in the past, and failed miserably every single time--all I want to do is use it as a gateway (ie, make the proxy invisible to the application) for e.g. apt packages, so I just ended up using apt-cache or whatever other appropriate software, but I'd far rather use something generic that just works on 90% of the software I use at home, whether it's reading webcomics or repeatedly installing the same software in a dozen VMs with slightly different configurations, or even just browsing remote filesystems via webdav.
- megous 6y ago
  I use nginx to proxy cache the Arch Linux package repository transparently. It's fairly easy to set up, and enables nice features like contacting a secondary mirror if the first one is down, or when multiple requests hit the same resource, all are blocked waiting for a single merged package download, so the proxy will not make the download multiple times if I run pacman -Syu on my 18 machines in parallel. And it's all just 20-30 lines of nginx config. It's not transparent though.
- EnigmaCurry 6y ago
  this is my way: https://github.com/EnigmaCurry/lazy-distro-mirrors https://github.com/EnigmaCurry/lazy-distro-mirrors
swuecho 6y ago
https://docs.docker.com/registry/ https://docs.docker.com/registry/ you can set it up in less 10 min and the only thing required is to add '--insecure-registry' in your client. It is not a issue if all your machine are in private network.
- johnmaguire2013 6y ago
  Isn't there no authentication on that registry? I guess that's fine if you don't believe in zero-trust architecture.
  - swuecho 6y ago
    you are right. That is what you can get in minutes.
    - majewsky 6y ago
      If you cannot get a TLS cert for internal infrastructure in a few minutes, I'd recommend you start looking into why.
      - swuecho 6y ago
        no good document on it and it is not very important for me ( I run it on homelab). still wonder how to do it in minutes.
        Aeolos 6y ago
        I use this (in a docker image) to generate certificates automatically: https://github.com/adferrand/dnsrobocert https://github.com/adferrand/dnsrobocert Expect to spend 1-2 hours first time you try it until you can setup the correct DNS records, API keys and configuration. Afterwards it's pretty hands off, every three months you'll receive an email from letsencrypt and you'll have to rerun this script to regenerate your certificates. Takes 2-3 minutes max (but of course you still need to distribute your certificates to all relevant services...)
Sherl 6y ago
If you run on Kuberenetes, the image is/can be cached at network layer.

sytse 6y ago

If you want to run a local registry to stay below the 100 pulls per 6 hours limit please consider GitLab. The Dependency Proxy https://docs.gitlab.com/ee/user/packages/dependency_proxy/ https://docs.gitlab.com/ee/user/packages/dependency_proxy/ will cache docker images. This way you stay within the limits Docker set and subsequent pulls should be faster as well.

oppositelock 6y ago

I run tens of thousands of docker images in production, or rather, tens of thousands of copies of a few hundred images. If you do something like this, you absolutely MUST have a local registry. Harbor [1], JFrog [2], and Quay [3] would be the first ones that I look at. Harbor is open source, free, and a member of the CNCF. You will need to do a little bit of work to set it up to scale properly. JFrog offers a SaaS registry, but you will pay big $$ based on pull traffic. Their commercial site license is about $3k/year. Quay is older than either of them, stable, and high quality. I'd start with Harbor these days. [1] https://goharbor.io/ https://goharbor.io/ [2] https://www.jfrog.com/confluence/display/JFROG/JFrog+Artifactory https://www.jfrog.com/confluence/display/JFROG/JFrog+Artifac... [3] https://quay.io/ https://quay.io/

drzaiusx11 6y ago
I was a happy user of JFrog's registries via site license at my last 2 places. Seemed to just work as expected. Didn't have visibility into the cost though (other teams set it up) so I had no idea it was $3k/year.
manquer 6y ago
Just to add all the major cloud service providers provide registries ACR /ECR/GCR etc . If you run k8s service with one of the them in my experience it is best to use the corresponding registry. I have pulled and run 20k times a 1GB image in less than 10-15 minutes without breaking a sweat. Finally GitHub packages offers a registry out of the box . It is great for CI and devs to access . I generally have the tags mirrored from tags GitHub for production to ACR .
- ztjio 6y ago
  Looking at doing GitHub Packages for direct-to-dev and mirroring into ECR over here. Seems sound. But also considering other options as ECR is a pain to work with. That said, word of warning for anyone looking at GitHub Packages for docker registry: it's broken with containerd and some other similar tools. They (GitHub) are currently working on a fix: https://github.com/containerd/containerd/issues/3291 https://github.com/containerd/containerd/issues/3291
  - lolinder 6y ago
    I got set up with ECR without any difficulties whatsoever. You do have to authenticate before pulls and pushes, but that can be scripted very easily.
- threeseed 6y ago
  Github Docker Registry is a mess and should be avoided at all costs. 1) It is broken and unusable on Kubernetes and Docker Swarm. 2) It is flaky often returning 500 type errors. 3) It is expensive as the amount of pull bandwidth is very limited.
  - manquer 6y ago
    Github packages works with Github CI out of the box, it makes development lot easier, like I mentioned for best networking in prod you should always use the registry from your k8s Provider, mirroring the Github registry to ECR/GCR/ACR is fairly straightforward. Bandwidth costs are eliminated, network is lot more reliable intra DC.
apple4ever 6y ago
We have not had good luck with Quay. They are not stable, especially as of late. There was a period last month where for two weeks pulling images was a crapshoot.
hamiltont 6y ago
Thank you very much. This is exactly the type of info I needed.

mschuster91 6y ago

Just drop a Sonatype Nexus instance on a Docker container somewhere on your network. Alternatively, use Squid if you don't push to the public Docker registry, although you might need to mess around with internal CA for SSL...

mvanbaak 6y ago
Nexus in a container... because storage in containers is such a good idea? Any vps with a disk is probably a better idea
- 1337shadow 6y ago
  You can still bind mount a directory into a container...
- Spivak 6y ago
  Storage in containers has been a long solved issue. The defaults are unfortunate because but make sense for ease of use. Your container root should be read only, ephemeral storage lives in a tmpfs or dynamic volumes depending on performance and size needs, and persistent storage lives in volumes.
efrecon 6y ago
I would stay away from Nexus. It has problems with latest tags.
Spivak 6y ago
Docker supports proxies (they call them “pullthrough repos”) so you don’t have to be so generic as an http proxy.

drzaiusx11 6y ago

I just use ECR[1] which in many cases costs less and is fully locked down behind my AWS VPC With ECR you pay for image storage: $0.09 per GB after the first 1 GB which is free [1] https://aws.amazon.com/ecr/ https://aws.amazon.com/ecr/

tuananh 6y ago
are you gonna rebuild all the images that you use and push to ECR?
- drzaiusx11 6y ago
  nope. you don't have to rebuild images to push to different registries pull from docker hub once, push to ECR. then pull from ECR as much as wish

efreak 6y ago

Personally, I wish generic caching proxies were still a thing, and easier to set up. I've tried setting up squid several times in the past, and failed miserably every single time--all I want to do is use it as a gateway (ie, make the proxy invisible to the application) for e.g. apt packages, so I just ended up using apt-cache or whatever other appropriate software, but I'd far rather use something generic that just works on 90% of the software I use at home, whether it's reading webcomics or repeatedly installing the same software in a dozen VMs with slightly different configurations, or even just browsing remote filesystems via webdav.

megous 6y ago
I use nginx to proxy cache the Arch Linux package repository transparently. It's fairly easy to set up, and enables nice features like contacting a secondary mirror if the first one is down, or when multiple requests hit the same resource, all are blocked waiting for a single merged package download, so the proxy will not make the download multiple times if I run pacman -Syu on my 18 machines in parallel. And it's all just 20-30 lines of nginx config. It's not transparent though.
EnigmaCurry 6y ago
this is my way: https://github.com/EnigmaCurry/lazy-distro-mirrors https://github.com/EnigmaCurry/lazy-distro-mirrors

swuecho 6y ago

https://docs.docker.com/registry/ https://docs.docker.com/registry/ you can set it up in less 10 min and the only thing required is to add '--insecure-registry' in your client. It is not a issue if all your machine are in private network.

johnmaguire2013 6y ago
Isn't there no authentication on that registry? I guess that's fine if you don't believe in zero-trust architecture.
- swuecho 6y ago
  you are right. That is what you can get in minutes.
  - majewsky 6y ago
    If you cannot get a TLS cert for internal infrastructure in a few minutes, I'd recommend you start looking into why.
    - swuecho 6y ago
      no good document on it and it is not very important for me ( I run it on homelab). still wonder how to do it in minutes.
      - Aeolos 6y ago
        I use this (in a docker image) to generate certificates automatically: https://github.com/adferrand/dnsrobocert https://github.com/adferrand/dnsrobocert Expect to spend 1-2 hours first time you try it until you can setup the correct DNS records, API keys and configuration. Afterwards it's pretty hands off, every three months you'll receive an email from letsencrypt and you'll have to rerun this script to regenerate your certificates. Takes 2-3 minutes max (but of course you still need to distribute your certificates to all relevant services...)

Sherl 6y ago

If you run on Kuberenetes, the image is/can be cached at network layer.