4 ms·

Anyone have recommendations for an image cache? Native kubernetes a plus. What would be really nice is a system with mutating admission webhooks for pods which

Anyone have recommendations for an image cache? Native kubernetes a plus. What would be really nice is a system with mutating admission webhooks for pods which kicks off a job to mirror the image to a local registry and then replaces the image reference with the mirrored location.

andrewstuart2 1y ago
CNCF has harbor [0], which I use at home and have deployed in a few clusters at work, and it works well as a pull through cache. In /etc/containers/registries.conf it's just another line below any registry you want mirrored. [[registry]] location = "docker.io" [[registry.mirror]] location = "core.yourharbor.example.com/hub" Where hub is the name of the proxy you configured for, in this case, docker.io. It's not quite what you're asking for but it can definitely be transparent to users. I think the bonus is that if you look at a podspec it's obvious where the image originates and you can pull it yourself on your machine, versus if you've mutated the podspec, you have to rely on convention. [0] https://goharbor.io/ https://goharbor.io/
- alias_neo 1y ago
  I would add, for anyone not familiar with it, that this (and more advanced mirroring, etc) is just as easily done from the really nice Web UI (if that's your cup of tea).
  - andrewstuart2 1y ago
    Yeah, to clarify, I had to first set up /hub as a caching proxy using the UI. Then the above configuration change was what I needed on my nodes in order to transparently use the proxy without changing podspecs.
- esseph 1y ago
  Seconding harbor
da768 1y ago
Not Google Artifact Registry... Our Docker Hub pull-through mirror went down with the Docker Hub outage. Images were still there but all image tags were gone
- gjvr 1y ago
  Thanks for sharing. Good to know. I Was considering using it for that purpose.
edoceo 1y ago
We do a local (well, internal) mirror for "all" these things. So, we're basically never stuck. It mirrors our CPAN, NPM, Composer, Docker and other of these web-repos. Helps on the CI tooling as well.
- baq 1y ago
  This is the way. At some point it’s way too expensive for a single repo in your supply chain to go down or even pull a package.
- goku12 1y ago
  What server do you use?
  - edoceo 1y ago
    Some hacked together homemade jank. I have a dream of refining and releasing.
    - goku12 1y ago
      I haven't seen anything like it, except as part of Gitlab perhaps. But I'm sure that it would have a lot of demand and has the potential to become a popular project. Good luck!
NickHirras 1y ago
I've been using Amazon ECR as an alternative source. https://gallery.ecr.aws/ https://gallery.ecr.aws/
vitaliyf 1y ago
https://github.com/spegel-org/spegel https://github.com/spegel-org/spegel
- __turbobrew__ 1y ago
  That looks pretty close to what I want. Thanks!
  - phillebaba 1y ago
    I initially built Spegel to deal with a series of GHCR outages along with rate limit changes in Docker Hub. I am a bit biased but it is a no brainier to run Spegel to mitigate these types of issues.
    - __turbobrew__ 1y ago
      Thank you. Question, how large of clusters have you deployed spegel to? I run clusters which are well beyond the defined k8s limits and sometimes when we put stuff in there it blows up as it wasn’t ever tested at those scales (for example Cilium) https://kubernetes.io/docs/setup/best-practices/cluster-large/ https://kubernetes.io/docs/setup/best-practices/cluster-larg...
VonGuard 1y ago
Quay.io
- hoherd 1y ago
  Quay.io goes down way too frequently to be a solution to Docker Hub being down.
tfolbrecht 1y ago
I usually do upstream image mirroring as part of CI. Registries are built into GitLab, AWS (ECR), GitHub, etc
issei 1y ago
I've been using https://github.com/enix/kube-image-keeper https://github.com/enix/kube-image-keeper on some of my clusters - it is a local docker registry running on cluster, with a proxy and mutation webhooks. I also evaluated spegel, but currently it isn't possible to setup on GKE
lars_francke 1y ago
I'm using a different approach for local testing where I don't want to redownload images over and over: https://github.com/stackabletech/k8s-local-dev https://github.com/stackabletech/k8s-local-dev Basically it's a k3s configured to use a local mirror and that local mirror is running the Zot registry (https://zotregistry.dev/v2.1.8/ https://zotregistry.dev/v2.1.8/). It is configured to automatically expired old images so my local hard drive isn't filled up).
alias_neo 1y ago
Depending on what other (additional) features you're willing to accept, the GoHarbor[0] registry supports pull-through as well as mirroring and other features, it's a nice registry that also supports other OCI stuff like Helm charts, and does vulnerability scanning with "Interrogation Services" like Trivy. I've been using it at home and work for a few years now, might be a bit overkill if you just want a simple registry, but is a really nice tool for anyone who can benefit from the other features. [0] https://goharbor.io/ https://goharbor.io/
philipallstar 1y ago
You can use Artifactory as a "front" for a variety of registries, including Docker, so it'll pull once and then use its cached image.

andrewstuart2 1y ago

CNCF has harbor [0], which I use at home and have deployed in a few clusters at work, and it works well as a pull through cache. In /etc/containers/registries.conf it's just another line below any registry you want mirrored. [[registry]] location = "docker.io" [[registry.mirror]] location = "core.yourharbor.example.com/hub" Where hub is the name of the proxy you configured for, in this case, docker.io. It's not quite what you're asking for but it can definitely be transparent to users. I think the bonus is that if you look at a podspec it's obvious where the image originates and you can pull it yourself on your machine, versus if you've mutated the podspec, you have to rely on convention. [0] https://goharbor.io/ https://goharbor.io/

alias_neo 1y ago
I would add, for anyone not familiar with it, that this (and more advanced mirroring, etc) is just as easily done from the really nice Web UI (if that's your cup of tea).
- andrewstuart2 1y ago
  Yeah, to clarify, I had to first set up /hub as a caching proxy using the UI. Then the above configuration change was what I needed on my nodes in order to transparently use the proxy without changing podspecs.
esseph 1y ago
Seconding harbor

da768 1y ago

Not Google Artifact Registry... Our Docker Hub pull-through mirror went down with the Docker Hub outage. Images were still there but all image tags were gone

gjvr 1y ago
Thanks for sharing. Good to know. I Was considering using it for that purpose.

edoceo 1y ago

We do a local (well, internal) mirror for "all" these things. So, we're basically never stuck. It mirrors our CPAN, NPM, Composer, Docker and other of these web-repos. Helps on the CI tooling as well.

baq 1y ago
This is the way. At some point it’s way too expensive for a single repo in your supply chain to go down or even pull a package.
goku12 1y ago
What server do you use?
- edoceo 1y ago
  Some hacked together homemade jank. I have a dream of refining and releasing.
  - goku12 1y ago
    I haven't seen anything like it, except as part of Gitlab perhaps. But I'm sure that it would have a lot of demand and has the potential to become a popular project. Good luck!

NickHirras 1y ago

I've been using Amazon ECR as an alternative source. https://gallery.ecr.aws/ https://gallery.ecr.aws/

vitaliyf 1y ago

https://github.com/spegel-org/spegel https://github.com/spegel-org/spegel

__turbobrew__ 1y ago
That looks pretty close to what I want. Thanks!
- phillebaba 1y ago
  I initially built Spegel to deal with a series of GHCR outages along with rate limit changes in Docker Hub. I am a bit biased but it is a no brainier to run Spegel to mitigate these types of issues.
  - __turbobrew__ 1y ago
    Thank you. Question, how large of clusters have you deployed spegel to? I run clusters which are well beyond the defined k8s limits and sometimes when we put stuff in there it blows up as it wasn’t ever tested at those scales (for example Cilium) https://kubernetes.io/docs/setup/best-practices/cluster-large/ https://kubernetes.io/docs/setup/best-practices/cluster-larg...

VonGuard 1y ago

Quay.io

hoherd 1y ago
Quay.io goes down way too frequently to be a solution to Docker Hub being down.

tfolbrecht 1y ago

I usually do upstream image mirroring as part of CI. Registries are built into GitLab, AWS (ECR), GitHub, etc

issei 1y ago

I've been using https://github.com/enix/kube-image-keeper https://github.com/enix/kube-image-keeper on some of my clusters - it is a local docker registry running on cluster, with a proxy and mutation webhooks. I also evaluated spegel, but currently it isn't possible to setup on GKE

lars_francke 1y ago

I'm using a different approach for local testing where I don't want to redownload images over and over: https://github.com/stackabletech/k8s-local-dev https://github.com/stackabletech/k8s-local-dev Basically it's a k3s configured to use a local mirror and that local mirror is running the Zot registry (https://zotregistry.dev/v2.1.8/ https://zotregistry.dev/v2.1.8/). It is configured to automatically expired old images so my local hard drive isn't filled up).

alias_neo 1y ago

Depending on what other (additional) features you're willing to accept, the GoHarbor[0] registry supports pull-through as well as mirroring and other features, it's a nice registry that also supports other OCI stuff like Helm charts, and does vulnerability scanning with "Interrogation Services" like Trivy. I've been using it at home and work for a few years now, might be a bit overkill if you just want a simple registry, but is a really nice tool for anyone who can benefit from the other features. [0] https://goharbor.io/ https://goharbor.io/

philipallstar 1y ago

You can use Artifactory as a "front" for a variety of registries, including Docker, so it'll pull once and then use its cached image.