7 ms·

I suspect the push for podman was more about how docker ignored CGroupsV2 for so long that Fedora eventually turned it on anyway which broke docker and then tol

I suspect the push for podman was more about how docker ignored CGroupsV2 for so long that Fedora eventually turned it on anyway which broke docker and then told users to switch to podman.

cozzyd 5y ago
And EL8 for that matter
kbenson 5y ago
I think a really big part of it was where Red Hat asked Docker to accept their patch that allowed people to run docker with local registries only (no docker.io), and were told Docker would not be accepting that patch, and to go pound sand if they didn't like it (eh, so maybe not so forcefully). The first thing I tried to figure out when looking into Docker for work was how to limit the registries it would look at to only be our own when used in production, and I was surprised to find out you can't (at least not without a hack to make it think it's using a mirror and just hitting your registry first).
- intsunny 5y ago
  Fascinating history, is there a PR link for this exchange between Redhat and Docker?
  - SSLy 5y ago
    https://github.com/moby/moby/issues/1988#issuecomment-271402422 https://github.com/moby/moby/issues/1988#issuecomment-271402...
  - kbenson 5y ago
    The main ones I know of are https://github.com/moby/moby/pull/10411 https://github.com/moby/moby/pull/10411 and https://github.com/moby/moby/issues/11816 https://github.com/moby/moby/issues/11816 There's a comment in the second link that's references in the first one that explains the rationale for the pull request refusal: Like pointed out earlier (#11815), this would fragment the namespace, and hurt the community pretty badly, making dockerfiles no longer portable. You can see that full comment at https://github.com/moby/moby/issues/11816#issuecomment-86732779 https://github.com/moby/moby/issues/11816#issuecomment-86732..., and it also references that this will be possible with signed images, so I don't know what happened after that (but that was over 6 years ago).
- junon 5y ago
  Docker does this a lot. For example, we were trying to turn off gzipping images on the wire when pulling because it actually cost more when done from the intranet. You can't. And modifying the source was so convoluted that we gave up. Then we needed to clean up docker (before there were commands to do that) when it started to eat up all of the disk space. To our (un)surprise, Docker uses 3 (!!) different storage formats, many of them having redundant information, and editing one of them would cause the other to be corrupted. One was a binary database format that was specific to Go and didn't have any utility CLI to work with, so you had to write a programmatic interface with it just to edit it. Or how about the fact that even if you issue commands directly to docker over the HTTP Unix socket, it will deadlock if you issue too many commands to it? This became our nightmare when trying to implement one of the first iterations of custom deployment backends at ZEIT. In fact, the entire project failed because of docker (there was no great alternative at the time).
  - wernerb 5y ago
    I am honestly wondering more about your specific use case. Does the CPU cost outweigh the storage cost? Is it a timing problem (speedup) or are you at such large scale (on premise)? Maybe where I'm getting at is, I can think of 99 problems but docker gzip ain't one :) how was this a priority (at some point)
    - infogulch 5y ago
      Pretty sure it's CPU vs network not CPU vs storage. A fast internal network can be cheap compared to the cpu required to pack/unpack images on every download. I wonder if choosing a fast-to-decompress algorithm like zstd would change that.
    - junon 5y ago
      As we were a deployment company (ZEIT) our usecase was quite different. And yes, as the other person mentioned, it was on the wire GZIP, not storage concerns.
- infogulch 5y ago
  I agree that it should be possible to disable the default registry, but I'm not sure I agree with allowing you to override it. (These requests appear to be conflated in various comments.) Use your own registry by specifying the domain first `myregistry.example.com/repo/image`; an unadorned `repo/image` being globally reserved as shorthand for `registry.docker.io/repo/image` seems fine. Allowing overriding the meaning of `repo/image` would be a support nightmare for both moby and internal IT, just use qualified names.
  - light_hue_1 5y ago
    This suggestion is a security nightmare! Anyone is one typo away from installing random junk from the internet on your machines. No one should be using docker in production while it can connect to a public registry where you have zero control of its contents.
    - infogulch 5y ago
      Did you read my comment? Because I can't find any interpretation of your response that makes sense assuming you comprehend the actual content of my statements.
      - light_hue_1 5y ago
        I did read the comment. And I comprehended the content of the statements. And I'm scared that this obvious security risk isn't horrifying to you. > I agree that it should be possible to disable the default registry, but I'm not sure I agree with allowing you to override it. (These requests appear to be conflated in various comments.) Use your own registry by specifying the domain first `myregistry.example.com/repo/image`; an unadorned `repo/image` being globally reserved as shorthand for `registry.docker.io/repo/image` seems fine. Allowing overriding the meaning of `repo/image` would be a support nightmare for both moby and internal IT, just use qualified names. Literally anyone in your company can forget to say `myregistry.example.com/` at any moment. And then your whole infrastructure runs on some random image that you didn't vet. You're a typo away from having your machines owned, your entire infrastructure falling over, your data being exposed to the anyone. This is no way to live and it's no way to run a company.
infogulch 5y ago
I hadn't heard about CGroupsV2. Apparently it's been in the kernel since 2015. Seems like a better design. https://thenewstack.io/linux-cgroups-v2-brings-rootless-containers-superior-memory-management/ https://thenewstack.io/linux-cgroups-v2-brings-rootless-cont... https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2...
- foxpurple 5y ago
  Every distro had it turned off for ages because turning it on would break docker. So eventually fedora decided docker was never going to added it and turned it on anyway. Then shortly after, docker adds support
  - vetinari 5y ago
    Between fedora and docker, it didn't help, that docker made a first release for current fedora version always about a month before fedora was going to make a new release (i.e. with fedora's 6 month release cycle, the corresponding docker release for that fedora version was 5 months late).

I suspect the push for podman was more about how docker ignored CGroupsV2 for so long that Fedora eventually turned it on anyway which broke docker and then told users to switch to podman.

cozzyd 5y ago

And EL8 for that matter

kbenson 5y ago

I think a really big part of it was where Red Hat asked Docker to accept their patch that allowed people to run docker with local registries only (no docker.io), and were told Docker would not be accepting that patch, and to go pound sand if they didn't like it (eh, so maybe not so forcefully). The first thing I tried to figure out when looking into Docker for work was how to limit the registries it would look at to only be our own when used in production, and I was surprised to find out you can't (at least not without a hack to make it think it's using a mirror and just hitting your registry first).

intsunny 5y ago
Fascinating history, is there a PR link for this exchange between Redhat and Docker?
- SSLy 5y ago
  https://github.com/moby/moby/issues/1988#issuecomment-271402422 https://github.com/moby/moby/issues/1988#issuecomment-271402...
- kbenson 5y ago
  The main ones I know of are https://github.com/moby/moby/pull/10411 https://github.com/moby/moby/pull/10411 and https://github.com/moby/moby/issues/11816 https://github.com/moby/moby/issues/11816 There's a comment in the second link that's references in the first one that explains the rationale for the pull request refusal: Like pointed out earlier (#11815), this would fragment the namespace, and hurt the community pretty badly, making dockerfiles no longer portable. You can see that full comment at https://github.com/moby/moby/issues/11816#issuecomment-86732779 https://github.com/moby/moby/issues/11816#issuecomment-86732..., and it also references that this will be possible with signed images, so I don't know what happened after that (but that was over 6 years ago).
junon 5y ago
Docker does this a lot. For example, we were trying to turn off gzipping images on the wire when pulling because it actually cost more when done from the intranet. You can't. And modifying the source was so convoluted that we gave up. Then we needed to clean up docker (before there were commands to do that) when it started to eat up all of the disk space. To our (un)surprise, Docker uses 3 (!!) different storage formats, many of them having redundant information, and editing one of them would cause the other to be corrupted. One was a binary database format that was specific to Go and didn't have any utility CLI to work with, so you had to write a programmatic interface with it just to edit it. Or how about the fact that even if you issue commands directly to docker over the HTTP Unix socket, it will deadlock if you issue too many commands to it? This became our nightmare when trying to implement one of the first iterations of custom deployment backends at ZEIT. In fact, the entire project failed because of docker (there was no great alternative at the time).
- wernerb 5y ago
  I am honestly wondering more about your specific use case. Does the CPU cost outweigh the storage cost? Is it a timing problem (speedup) or are you at such large scale (on premise)? Maybe where I'm getting at is, I can think of 99 problems but docker gzip ain't one :) how was this a priority (at some point)
  - infogulch 5y ago
    Pretty sure it's CPU vs network not CPU vs storage. A fast internal network can be cheap compared to the cpu required to pack/unpack images on every download. I wonder if choosing a fast-to-decompress algorithm like zstd would change that.
  - junon 5y ago
    As we were a deployment company (ZEIT) our usecase was quite different. And yes, as the other person mentioned, it was on the wire GZIP, not storage concerns.
infogulch 5y ago
I agree that it should be possible to disable the default registry, but I'm not sure I agree with allowing you to override it. (These requests appear to be conflated in various comments.) Use your own registry by specifying the domain first `myregistry.example.com/repo/image`; an unadorned `repo/image` being globally reserved as shorthand for `registry.docker.io/repo/image` seems fine. Allowing overriding the meaning of `repo/image` would be a support nightmare for both moby and internal IT, just use qualified names.
- light_hue_1 5y ago
  This suggestion is a security nightmare! Anyone is one typo away from installing random junk from the internet on your machines. No one should be using docker in production while it can connect to a public registry where you have zero control of its contents.
  - infogulch 5y ago
    Did you read my comment? Because I can't find any interpretation of your response that makes sense assuming you comprehend the actual content of my statements.
    - light_hue_1 5y ago
      I did read the comment. And I comprehended the content of the statements. And I'm scared that this obvious security risk isn't horrifying to you. > I agree that it should be possible to disable the default registry, but I'm not sure I agree with allowing you to override it. (These requests appear to be conflated in various comments.) Use your own registry by specifying the domain first `myregistry.example.com/repo/image`; an unadorned `repo/image` being globally reserved as shorthand for `registry.docker.io/repo/image` seems fine. Allowing overriding the meaning of `repo/image` would be a support nightmare for both moby and internal IT, just use qualified names. Literally anyone in your company can forget to say `myregistry.example.com/` at any moment. And then your whole infrastructure runs on some random image that you didn't vet. You're a typo away from having your machines owned, your entire infrastructure falling over, your data being exposed to the anyone. This is no way to live and it's no way to run a company.

infogulch 5y ago

I hadn't heard about CGroupsV2. Apparently it's been in the kernel since 2015. Seems like a better design. https://thenewstack.io/linux-cgroups-v2-brings-rootless-containers-superior-memory-management/ https://thenewstack.io/linux-cgroups-v2-brings-rootless-cont... https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2...

foxpurple 5y ago
Every distro had it turned off for ages because turning it on would break docker. So eventually fedora decided docker was never going to added it and turned it on anyway. Then shortly after, docker adds support
- vetinari 5y ago
  Between fedora and docker, it didn't help, that docker made a first release for current fedora version always about a month before fedora was going to make a new release (i.e. with fedora's 6 month release cycle, the corresponding docker release for that fedora version was 5 months late).