5 ms·

That's great news. What do you consider a "properly implemented patch"? I would think the most bulletproof & simplest patch would simply allow the default ind

That's great news. What do you consider a "properly implemented patch"? I would think the most bulletproof & simplest patch would simply allow the default index (index.docker.io) to be reconfigured to something else in docker.conf. Would you support a patch that did that? Edit: And perhaps https://registry-1.docker.io/v2/ https://registry-1.docker.io/v2/ as well?

shykes 12y ago
I think the best way is to allow a "whitelist mode" where only an explicitly specified list of URLs are allowed to be reached. Everything else would be blocked by default. This should give ops the peace of mind they need. Note that this is an ACL change, and not a namespace change. That is important because we want image names to have the same meaning everywhere, regardless of site-specific configuration. So for example, "docker pull ubuntu" should always mean "install ubuntu from the official docker library". This is crucial to the developer experience and to respect the principle of separation of concern between dev and ops. However, if ops chooses to block access to the standard library then "docker pull ubuntu" will fail with "access blocked by your administrator", which is totally acceptable. What we don't want is the operation silently substituting a site-specific image, without the knowledge of the end user, thereby breaking their build in a thousand invisible ways. I hope this helps. Does this mean I should look forward to a patch from you? :)
- justinsb 12y ago
  I will happily provide a patch that implements the desired functionality :-) However, I think people want the option of not using the docker registry. It is not under their control, it is slow, and it is a single point of failure. Having to recompile Docker to remove the magic constants is painful. I can see that the ACL functionality would be useful, but I don't think it addresses those concerns. Even if you believe it is a bad idea to allow users to change the meaning of image names, you should allow users the freedom to choose their own path. I think this is a fairly basic tenet of open-source: that true innovation comes when people are allowed to try new things, even if they look like a bad idea at the time.
  - shykes 12y ago
    I totally agree that users should be given the freedom to choose their own path. For example that's why we added pluggable backends for storage and sandboxing very early, before even shipping 1.0 :) At the same time, I care a lot about design and usability - and good design requires constraints. So there is a fundamental tension between flexibility and usability. My approach to this problem is to put usability first, and improve flexibility over time. The rationale is simple: you can always make things more flexible over time - but once usability is broken, it's almost impossible to fix. So, to apply this in the topic at hand: * Usability first: no matter what, don't break the predictable meaning of image names. * Flexibility over time, step 0: allow admins to block access to the Hub. Since the standard library is only hosted on the Hub, this blocks access to the standard library. Note that you are not at all required to use the Hub. You can refer to any other registry, today, simply by "docker pull URL". * Flexibility over time, step 1: allow hosting the standard library outside the Hub. The standard library is actually community project, very similar to Homebrew. The fact that it can only be hosted on the Hub is a technical limitation. We should lift that limitation, so that even if you block access to the Hub, you can still access the standard library. This could be via a private mirror, community-controller mirrors, or perhaps a bittorrent transport :) * Flexibility over time, step 2: allow complete control over which images are downloaded from where. This means completely separating the naming of images from their transport. This allows a sysadmin to compose an image storage and transport setup that fits their needs: S3 for this namespace, high-performance NFS filer for that, public bittorrent transport for this authorized subset of public images, Hub for the the QA department, etc. The key technical requirement here is to make image self-describing, and make cryptographic signature and verification mandatory.
    - vidarh 12y ago
      > * Usability first: no matter what, don't break the predictable meaning of image names. Give up any idea that you have control over this. You don't. As a user, I don't want you to have control over this, because it is directly contrary to my interests to not be able to redirect requests for "docker pull ubuntu" as per your example, to go somewhere I have full control over, so that I can choose to get exactly the image I expect matches the "ubuntu" image name rather than whatever the index will currently return for that name, and do so without having to change every reference to that name to something relative to my own registry. You can make this hard, and have it continue to annoy users, or you can fix this, but this is a the biggest usability problem with Docker to me the way it stands now. I find it hard to take you seriously when you claim usability first, and then claim that there is predictable meaning of image names: That is only true if you by "predictable meaning of image names" means "I don't really know what I'll get". This is especially true because of the lax use of tags all over the place. The irony is that your "key technical requirement" for "flexibility over time, step 2" is far more important with the current situation than in actually would have been if we had the flexibility you describe for that step: If Docker provided that flexibility today, then I could have easy complete control over exactly which images gets accessed if I wanted to. Instead I need to resort to firewall rules and/or DNS hacks or changing the source if I want to prevent accidentally pulling in images different than what I expect.
- tinco 12y ago
  > What we don't want is the operation silently substituting a site-specific image, without the knowledge of the end user, thereby breaking their build in a thousand invisible ways But this is not preventable in any way but explicitly pulling an image with a signing key. Who knows what index.docker.io resolves to or is serving?
- vidarh 12y ago
  > However, if ops chooses to block access to the standard library then "docker pull ubuntu" will fail with "access blocked by your administrator", which is totally acceptable. Actually, it isn't, at least in production, because it forces us to rebuild a lot of images that references the standard library, when a more reasonable approach would be to mirror them. The index/registry/image identity problem is by far the weakest part of Docker, and what appears most attractive with Rocket, in my opinion. There are pretty much zero cases where I, in my ops role, can allow production deployments to have access to the official Docker repository, because it opens the door to pulling in all kinds of stuff that has not been vetted (e.g. referencing the "latest" images, and having that change between dev signing something off and deployment), and it creates all kinds of obnoxious failure scenarios. At the same time, I don't want devs to have the hassle of having to repackage all the images to point them to our internal registries, when we could easily mirror the images that have been tested. So if there's no easy way to point the default somewhere else, what we'll resort to instead is increasingly adding firewall rules to block the official registry, coupled with DNS tweaks to make *.docker.io point where we want it, or patching the code. Or switch to Rocket once it gets more mature, if Docker continues to make custom image management more troublesome than necessary.
  - shykes 12y ago
    I totally agree that mirroring of official images should be easier, and right now it's an obstacle to easier production deployment. This is why it's important to have cryptographically signed, self-describing images. Then it becomes irrelevant where you download them from, and anyone could host a public or private mirror. I am 100% in favor of it and we are upgrading the registry system to allow it. Happy to chat more on #docker-dev.

shykes 12y ago

I think the best way is to allow a "whitelist mode" where only an explicitly specified list of URLs are allowed to be reached. Everything else would be blocked by default. This should give ops the peace of mind they need. Note that this is an ACL change, and not a namespace change. That is important because we want image names to have the same meaning everywhere, regardless of site-specific configuration. So for example, "docker pull ubuntu" should always mean "install ubuntu from the official docker library". This is crucial to the developer experience and to respect the principle of separation of concern between dev and ops. However, if ops chooses to block access to the standard library then "docker pull ubuntu" will fail with "access blocked by your administrator", which is totally acceptable. What we don't want is the operation silently substituting a site-specific image, without the knowledge of the end user, thereby breaking their build in a thousand invisible ways. I hope this helps. Does this mean I should look forward to a patch from you? :)

justinsb 12y ago
I will happily provide a patch that implements the desired functionality :-) However, I think people want the option of not using the docker registry. It is not under their control, it is slow, and it is a single point of failure. Having to recompile Docker to remove the magic constants is painful. I can see that the ACL functionality would be useful, but I don't think it addresses those concerns. Even if you believe it is a bad idea to allow users to change the meaning of image names, you should allow users the freedom to choose their own path. I think this is a fairly basic tenet of open-source: that true innovation comes when people are allowed to try new things, even if they look like a bad idea at the time.
- shykes 12y ago
  I totally agree that users should be given the freedom to choose their own path. For example that's why we added pluggable backends for storage and sandboxing very early, before even shipping 1.0 :) At the same time, I care a lot about design and usability - and good design requires constraints. So there is a fundamental tension between flexibility and usability. My approach to this problem is to put usability first, and improve flexibility over time. The rationale is simple: you can always make things more flexible over time - but once usability is broken, it's almost impossible to fix. So, to apply this in the topic at hand: * Usability first: no matter what, don't break the predictable meaning of image names. * Flexibility over time, step 0: allow admins to block access to the Hub. Since the standard library is only hosted on the Hub, this blocks access to the standard library. Note that you are not at all required to use the Hub. You can refer to any other registry, today, simply by "docker pull URL". * Flexibility over time, step 1: allow hosting the standard library outside the Hub. The standard library is actually community project, very similar to Homebrew. The fact that it can only be hosted on the Hub is a technical limitation. We should lift that limitation, so that even if you block access to the Hub, you can still access the standard library. This could be via a private mirror, community-controller mirrors, or perhaps a bittorrent transport :) * Flexibility over time, step 2: allow complete control over which images are downloaded from where. This means completely separating the naming of images from their transport. This allows a sysadmin to compose an image storage and transport setup that fits their needs: S3 for this namespace, high-performance NFS filer for that, public bittorrent transport for this authorized subset of public images, Hub for the the QA department, etc. The key technical requirement here is to make image self-describing, and make cryptographic signature and verification mandatory.
  - vidarh 12y ago
    > * Usability first: no matter what, don't break the predictable meaning of image names. Give up any idea that you have control over this. You don't. As a user, I don't want you to have control over this, because it is directly contrary to my interests to not be able to redirect requests for "docker pull ubuntu" as per your example, to go somewhere I have full control over, so that I can choose to get exactly the image I expect matches the "ubuntu" image name rather than whatever the index will currently return for that name, and do so without having to change every reference to that name to something relative to my own registry. You can make this hard, and have it continue to annoy users, or you can fix this, but this is a the biggest usability problem with Docker to me the way it stands now. I find it hard to take you seriously when you claim usability first, and then claim that there is predictable meaning of image names: That is only true if you by "predictable meaning of image names" means "I don't really know what I'll get". This is especially true because of the lax use of tags all over the place. The irony is that your "key technical requirement" for "flexibility over time, step 2" is far more important with the current situation than in actually would have been if we had the flexibility you describe for that step: If Docker provided that flexibility today, then I could have easy complete control over exactly which images gets accessed if I wanted to. Instead I need to resort to firewall rules and/or DNS hacks or changing the source if I want to prevent accidentally pulling in images different than what I expect.
tinco 12y ago
> What we don't want is the operation silently substituting a site-specific image, without the knowledge of the end user, thereby breaking their build in a thousand invisible ways But this is not preventable in any way but explicitly pulling an image with a signing key. Who knows what index.docker.io resolves to or is serving?
vidarh 12y ago
> However, if ops chooses to block access to the standard library then "docker pull ubuntu" will fail with "access blocked by your administrator", which is totally acceptable. Actually, it isn't, at least in production, because it forces us to rebuild a lot of images that references the standard library, when a more reasonable approach would be to mirror them. The index/registry/image identity problem is by far the weakest part of Docker, and what appears most attractive with Rocket, in my opinion. There are pretty much zero cases where I, in my ops role, can allow production deployments to have access to the official Docker repository, because it opens the door to pulling in all kinds of stuff that has not been vetted (e.g. referencing the "latest" images, and having that change between dev signing something off and deployment), and it creates all kinds of obnoxious failure scenarios. At the same time, I don't want devs to have the hassle of having to repackage all the images to point them to our internal registries, when we could easily mirror the images that have been tested. So if there's no easy way to point the default somewhere else, what we'll resort to instead is increasingly adding firewall rules to block the official registry, coupled with DNS tweaks to make *.docker.io point where we want it, or patching the code. Or switch to Rocket once it gets more mature, if Docker continues to make custom image management more troublesome than necessary.
- shykes 12y ago
  I totally agree that mirroring of official images should be easier, and right now it's an obstacle to easier production deployment. This is why it's important to have cryptographically signed, self-describing images. Then it becomes irrelevant where you download them from, and anyone could host a public or private mirror. I am 100% in favor of it and we are upgrading the registry system to allow it. Happy to chat more on #docker-dev.