4 ms·

Why is it so hard to have effective mirrors of images? Images are basically content addressed, it should be easy to create an ecosystem where when you first us

Why is it so hard to have effective mirrors of images? Images are basically content addressed, it should be easy to create an ecosystem where when you first use an image you pin a copy on a private cache, instead of shuffling hundreds of GB around to each leaf node of your clusters. This way cloud providers could still sell you a managed service that implements such a cache, but it would be used transparently instead of having to change your manifests and/or FROM stanzas. Netflix iirc has implemented a registry that uses IPFS. But talking to the right registry still requires that the image name includes the hostname of the registry. The docker daemon has a registry-mirrors option; you can already use that to point to https://mirror.gcr.io https://mirror.gcr.io . But you can use it only to cache the special "docker hub" images. This seems to be a self-inflicted problem. Docker the company possibly wanted to be in the critical path

ryan29 6y ago
AFAIK, the FQDN is used for connecting to the registry, but it's not passed to the registry. So when you pull `ecr.example.com/namespace/project` the daemon connects to `ecr.example.com`, but the registry only sees `namespace/project`. There was an old proposal [1] to pass the FQDN to the registry. That would allow for an auto-magical pull through cache / mirror combo along the lines of what you're asking about. For now I use a Nexus as a pull through cache, so `hub.example.com/library/image` is an image from Docker Hub where I want it to come through the cache. Anything that's important enough to have a local copy gets pulled/pushed into my private registry. I actually find the distinction between the cache (volatile) and the registry (non-volatile) to be useful. If I had to take a guess as to why Docker stubbornly clung to the current system it's that it's adhesive. Once you start using an alternate registry for base images, etc. it becomes really tough to swap to another one. The big mistake there is that instead of being the go to destination for public images, everyone is wholesale switching to the registry from their preferred cloud provider. 1. https://github.com/docker/distribution/issues/1620 https://github.com/docker/distribution/issues/1620
lgbr 6y ago
Artifactory does precisely this. It's particularly useful since large organizations want to minimize their risk of, for example, a Docker Hub outage preventing an emergency hotfix. In Artifactory you can setup 'remote repositories' which are then overlayed on top of your own repository within Artifactory. Docker provides the `registry-mirrors` setting which then will automatically redirect all pulls to Artifactory rather than Docker hub. There's no need to change your manifests and/or FROM stanzas.
- ithkuil 6y ago
  But it's not transparent. In all places where you use the image you have to change the name of the image. This means more templating, more complexity etc
  - richardwhiuk 6y ago
    You can configure the Docker client to use the artifactory remote by default.
    - ithkuil 6y ago
      can you elaborate on that? Are you talking about the "registry-mirrors" option of the docker daemon?
      - richardwhiuk 6y ago
        Yeah - if you set that then pulls for python:3.7 say will pull from your mirror instead.
        ithkuil 6y ago
        ok; so the problems with that are: 1. it requires a change on the docker daemon config. On many container environments like k8s it's hard to manipulate that setting. 2. it works only for the default "implicit" index registry, aka docker hub. Not all images are distributed from that place.
        richardwhiuk 6y ago
        1) You can configure docker's daemon.json file on Kubernetes in the same way as every other system running Docker (in fact, you can apply it with a daemonset if you want to). 2) Yes, every other image mandates where you should get it from - Docker Hub's are unique in that regard. Normally the registry the image is pulled from is configurable in the Helm chart.
neomantra 6y ago
I made a Terraform module that mirrors Docker images between two registries. I built it to deal with GCP infrastructure (private networking and Cloud Run). When we migrated to GitLab, we used it to coordinate external dependencies between Docker->GitLab registries, using GitLab CI and their Terraform support. [Internally built images are just pushed directly during CI.]. We declare the mirror state and Terraform makes it so. https://registry.terraform.io/modules/neomantra/mirror/docker https://registry.terraform.io/modules/neomantra/mirror/docke...
user5994461 6y ago
Exactly. Docker made it sure they were in the critical path. If you compare to Debian or Ubuntu packages, same use case, they have full support for mirror. When you run a debian VM in the cloud it's automatically connecting to the AWS debian mirror to pull packages. Everyone wins. Docker distribution was designed to keep docker the company in control, not to have mirrors. Naturally it's getting substituted by competitors because they have no other choice.
jacques_chester 6y ago
> Images are basically content addressed Not really. You can use digests to distinguish a version of a particular repository/image, but you cannot address anything with just the digest.
- ithkuil 6y ago
  That's why I said "basically". /v2/<name>/blobs/<digest> The spec is ambiguous about what to do if a blob with a matching digest is found in the registry but not in the repository namespace <name>. The layer deduplication section of the spec does state that the same blob MUST be shared between repositories. The manifest digest endpoint suffers from a similar ambiguity. In any case, all I wanted to say is that it would have been quite possible to implement docker registry as a purely content addressed store + a key/value layer that binds human readable names to it. It's almost, but not quite that.
  - jacques_chester 6y ago
    I don't think it's basically CAS at all. You need the name in advance before you can narrow it down with the digest.
    - ithkuil 6y ago
      My point is that it would have been trivial to define the registry protocol to just require the digest. Do you see any reason why a pure content addressed registry protocol could have not existed?
      - jacques_chester 6y ago
        No, I absolutely agree with you. It could have been a CAS design, but right it's not. It would take a V3 of the Registry API -- I commented on that in the DigitalOcean thread: https://news.ycombinator.com/item?id=24983077 https://news.ycombinator.com/item?id=24983077

ryan29 6y ago

AFAIK, the FQDN is used for connecting to the registry, but it's not passed to the registry. So when you pull `ecr.example.com/namespace/project` the daemon connects to `ecr.example.com`, but the registry only sees `namespace/project`. There was an old proposal [1] to pass the FQDN to the registry. That would allow for an auto-magical pull through cache / mirror combo along the lines of what you're asking about. For now I use a Nexus as a pull through cache, so `hub.example.com/library/image` is an image from Docker Hub where I want it to come through the cache. Anything that's important enough to have a local copy gets pulled/pushed into my private registry. I actually find the distinction between the cache (volatile) and the registry (non-volatile) to be useful. If I had to take a guess as to why Docker stubbornly clung to the current system it's that it's adhesive. Once you start using an alternate registry for base images, etc. it becomes really tough to swap to another one. The big mistake there is that instead of being the go to destination for public images, everyone is wholesale switching to the registry from their preferred cloud provider. 1. https://github.com/docker/distribution/issues/1620 https://github.com/docker/distribution/issues/1620

lgbr 6y ago

Artifactory does precisely this. It's particularly useful since large organizations want to minimize their risk of, for example, a Docker Hub outage preventing an emergency hotfix. In Artifactory you can setup 'remote repositories' which are then overlayed on top of your own repository within Artifactory. Docker provides the `registry-mirrors` setting which then will automatically redirect all pulls to Artifactory rather than Docker hub. There's no need to change your manifests and/or FROM stanzas.

ithkuil 6y ago
But it's not transparent. In all places where you use the image you have to change the name of the image. This means more templating, more complexity etc
- richardwhiuk 6y ago
  You can configure the Docker client to use the artifactory remote by default.
  - ithkuil 6y ago
    can you elaborate on that? Are you talking about the "registry-mirrors" option of the docker daemon?
    - richardwhiuk 6y ago
      Yeah - if you set that then pulls for python:3.7 say will pull from your mirror instead.
      - ithkuil 6y ago
        ok; so the problems with that are: 1. it requires a change on the docker daemon config. On many container environments like k8s it's hard to manipulate that setting. 2. it works only for the default "implicit" index registry, aka docker hub. Not all images are distributed from that place.
        richardwhiuk 6y ago
        1) You can configure docker's daemon.json file on Kubernetes in the same way as every other system running Docker (in fact, you can apply it with a daemonset if you want to). 2) Yes, every other image mandates where you should get it from - Docker Hub's are unique in that regard. Normally the registry the image is pulled from is configurable in the Helm chart.

neomantra 6y ago

I made a Terraform module that mirrors Docker images between two registries. I built it to deal with GCP infrastructure (private networking and Cloud Run). When we migrated to GitLab, we used it to coordinate external dependencies between Docker->GitLab registries, using GitLab CI and their Terraform support. [Internally built images are just pushed directly during CI.]. We declare the mirror state and Terraform makes it so. https://registry.terraform.io/modules/neomantra/mirror/docker https://registry.terraform.io/modules/neomantra/mirror/docke...

user5994461 6y ago

Exactly. Docker made it sure they were in the critical path. If you compare to Debian or Ubuntu packages, same use case, they have full support for mirror. When you run a debian VM in the cloud it's automatically connecting to the AWS debian mirror to pull packages. Everyone wins. Docker distribution was designed to keep docker the company in control, not to have mirrors. Naturally it's getting substituted by competitors because they have no other choice.

jacques_chester 6y ago

> Images are basically content addressed Not really. You can use digests to distinguish a version of a particular repository/image, but you cannot address anything with just the digest.

ithkuil 6y ago
That's why I said "basically". /v2/<name>/blobs/<digest> The spec is ambiguous about what to do if a blob with a matching digest is found in the registry but not in the repository namespace <name>. The layer deduplication section of the spec does state that the same blob MUST be shared between repositories. The manifest digest endpoint suffers from a similar ambiguity. In any case, all I wanted to say is that it would have been quite possible to implement docker registry as a purely content addressed store + a key/value layer that binds human readable names to it. It's almost, but not quite that.
- jacques_chester 6y ago
  I don't think it's basically CAS at all. You need the name in advance before you can narrow it down with the digest.
  - ithkuil 6y ago
    My point is that it would have been trivial to define the registry protocol to just require the digest. Do you see any reason why a pure content addressed registry protocol could have not existed?
    - jacques_chester 6y ago
      No, I absolutely agree with you. It could have been a CAS design, but right it's not. It would take a V3 of the Registry API -- I commented on that in the DigitalOcean thread: https://news.ycombinator.com/item?id=24983077 https://news.ycombinator.com/item?id=24983077