6 ms·

It didn't help them that they rejected the traditionally successful ways of monetizing open source software. Which is, selling support contracts to large corpo

by dralley 9mo ago

It didn't help them that they rejected the traditionally successful ways of monetizing open source software. Which is, selling support contracts to large corporate users. Corporate customers didn't like the security implications of the Docker daemon running as root, they wanted better sandboxing and management (cgroups v2), wanted to be able to run their own internal registries, didn't want to have docker trying to fight with systemd, etc. Docker was not interested (in the early years) in adopting cgroups v2 or daemonless / rootless operation, and they wanted everyone to pay to use Dockerhub on the public internet rather than running their own internal registries, so docker-cli didn't support alternate registries for a long long time. And it seemed like they disliked systemd for "ideological" reasons to an extent that they didn't make much effort to resolve the problems that would crop up between docker and systemd. Because Docker didn't want to build the product that corporate customers wanted to use, and didn't accept patches when Red Hat tried to get them implemented those features themselves, eventually Red Hat just went out and built up Podman, Quay, and the entire ecosystem of tooling that those corporate customers wanted themselves (and sold it to them). That was a bit of an own goal.

Normal_gaussian 9mo ago
yes; its really notable that corporates and other support companies (e.g. redhat) don't want to start down the path of NIH, and will go to significant efforts to avoid it. However, once they have done it, it is very hard to make them come back.
- PaulHoule 9mo ago
  I think the Star Wars problem was that instead of making the movies at a steady cadence they stretched it out too long.
anonymars 9mo ago
I can't help but see a parallel with some of the entertainment franchises in recent years (Star Wars, etc.) -- where a company seems to be allergic to taking money by giving people what they want, and instead insists on telling people what they should want and blaming them when they don't
cpuguy83 9mo ago
Absolutely none of this is true. Docker had support contracts (Docker EE... and trying to remember, docker-cs before that naming pivot?). Corporate customers do not care about any of the things you mentioned. I mean, maybe some, but in general no. That's not what corps think about. There was never "no interest" at Docker in cgv2 or rootless. Never. cgv2 early on was not useable. It lacked so much functionality that v1 had. It also didn't buy much, particularly because most Docker users aren't manually managing cgroups themselves. Docker literally sold a private registry product. It was the first thing Docker built and sold (and no, it was not late, it was very early on).
- FireBeyond 9mo ago
  I mean you can say that, but on the topic of rootless, regardless of "interest" at Docker, they did nothing about it. I was at Red Hat at the time, a PM in the BU that created podman, and Docker's intransigence on rootless was probably the core issue that led to podman's creation.
  - mikepurvis 9mo ago
    I've really appreciated RH's work both on podman/buildah and in the supporting infrastructure like the kernel that enables nesting, like using buildah to build an image inside a containerized CI runner. That said, I've been really surprised to not see more first class CI support for a repo supplying its own Dockerfile and being like "stage 1 is to rebuild the container", "stage two is a bunch of parallel tests running in instances of the container". In modern Dockerfiles it's pretty easy to avoid manual cache-busting by keying everything to a package manager lockfile, so it's annoying that the default CI paradigm is still "separate job somewhere that rebuilds a static base container on a timer".
    - FireBeyond 9mo ago
      Yeah, I've moved on from there, but I agree. There wasn't a lot of focus on the CI side of things beyond the stuff that ArgoCD was doing, and Shipwright (which isn't really CI/CD focused but did some stuff around the actual build progress, but really suffered failure to launch).
PaulHoule 9mo ago
When Docker was new I had a really bad ADSL connection (2Mbps) and couldn't ever stack up a containerized system properly because Dockerhub would time out. I did large downloads all the time, I used to download 25GB games for my game consoles for instance. I just had to use schedule them and use tools that could resume downloads. If I'd had a local docker hub I might have used docker but because I didn't it was dead to me.
nyrikki 9mo ago
Unfortunately even podman etc.. are still limited by OCIs decision to copy the Docker model. Crun just stamp couples security profiles as an example, so everything in the shared kernel that is namespace incompatible is enabled. This is why it is trivial to get in-auditable communication between pods on a host etc…
- ragall 9mo ago
  > Unfortunately even podman etc.. are still limited by OCIs decision to copy the Docker model. Which parts of the model are you referring to ?
  - nyrikki 9mo ago
    OCI Container Runtimes like OCI's runc are "container runtimes", so the runtime spec[2] Basically, docker started using lxc, but wanted a go native option, and wrote runc. If you look at [0] you can see how it actually instantiates the container. Here is a random blog that describes it fairly well [1] crun is the podman related project written in c, which is more efficient than the go based runc. You can try this even as the user nobody 65534:65534, but you may need to make some dirs, or set envs. Here is an example pulling an image with podman to make it easier, but you could just make an OCI spec bundle and run it: mkdir hello cd hello podman pull docker.io/hello-world podman export $(podman create hello-world) > hello-world.tar mkdir rootfs tar -C rootfs -xf hello-world.tar runc spec --rootless sed -i 's;"sh";"/hello";' config.json runc run container1 Hello from Docker! runc doesn't support any form of constraints like a bounding set on seccomp, selinux, apparmor, etc.. but it will apply profiles you pass it. Basically it fails open, and with the current state of apparmor and selinux it is trivial to bypass the minimal userns restrictions they place. Historically, before rootless containers this was less of an issue, because you had to be a privileged user to launch a container. But with the holes in the LSMs, no ability to set administrative bounding sets, and the reality that none of the defaults constrain risky kernel functionality like vsock, openat2 etc... there are a million ways to break netns isolation etc... Originally the docker project wanted to keep all the complexity of mutating LSM rules etc... in containerd. and they also fought even basic controls like letting an admin disable the `--privileged` flag at the daemon level. Unfortunately due to momentum, opinions, and friction in general, that means that now those container runtimes have no restrictions on callers, and cannot set reasonable defaults. Thus now we have to resort to teaching every person who launches a container to be perfect and disable everything, which they never do. If you run a k8s cluster with nodes on VMs, try this for example, if it doesn't error out, any pod can talk to any other pod on the node, with a protocol you aren't logging, and which has limited ability to log anyway. (if your k8s nodes are running systemd v256+ and you aren't using containerd which blocked vsock, but cri-o, podman, etc... don't (at least up to a couple of weeks ago) socat - VSOCK-LISTEN:3000 You can also play around with other af_families as IPX, Appletalk, etc... are all available by default, or see if you can use openat2 to use some file in /proc to break out. [0] https://manpages.debian.org/testing/runc/runc-spec.8.en.html https://manpages.debian.org/testing/runc/runc-spec.8.en.html [1] https://mkdev.me/posts/the-tool-that-really-runs-your-containers-deep-dive-into-runc-and-oci-specifications https://mkdev.me/posts/the-tool-that-really-runs-your-contai... [2] https://github.com/opencontainers/runtime-spec/blob/main/README.md https://github.com/opencontainers/runtime-spec/blob/main/REA...

by dralley 9mo ago

Normal_gaussian 9mo ago

yes; its really notable that corporates and other support companies (e.g. redhat) don't want to start down the path of NIH, and will go to significant efforts to avoid it. However, once they have done it, it is very hard to make them come back.

PaulHoule 9mo ago
I think the Star Wars problem was that instead of making the movies at a steady cadence they stretched it out too long.

anonymars 9mo ago

I can't help but see a parallel with some of the entertainment franchises in recent years (Star Wars, etc.) -- where a company seems to be allergic to taking money by giving people what they want, and instead insists on telling people what they should want and blaming them when they don't

cpuguy83 9mo ago

Absolutely none of this is true. Docker had support contracts (Docker EE... and trying to remember, docker-cs before that naming pivot?). Corporate customers do not care about any of the things you mentioned. I mean, maybe some, but in general no. That's not what corps think about. There was never "no interest" at Docker in cgv2 or rootless. Never. cgv2 early on was not useable. It lacked so much functionality that v1 had. It also didn't buy much, particularly because most Docker users aren't manually managing cgroups themselves. Docker literally sold a private registry product. It was the first thing Docker built and sold (and no, it was not late, it was very early on).

FireBeyond 9mo ago
I mean you can say that, but on the topic of rootless, regardless of "interest" at Docker, they did nothing about it. I was at Red Hat at the time, a PM in the BU that created podman, and Docker's intransigence on rootless was probably the core issue that led to podman's creation.
- mikepurvis 9mo ago
  I've really appreciated RH's work both on podman/buildah and in the supporting infrastructure like the kernel that enables nesting, like using buildah to build an image inside a containerized CI runner. That said, I've been really surprised to not see more first class CI support for a repo supplying its own Dockerfile and being like "stage 1 is to rebuild the container", "stage two is a bunch of parallel tests running in instances of the container". In modern Dockerfiles it's pretty easy to avoid manual cache-busting by keying everything to a package manager lockfile, so it's annoying that the default CI paradigm is still "separate job somewhere that rebuilds a static base container on a timer".
  - FireBeyond 9mo ago
    Yeah, I've moved on from there, but I agree. There wasn't a lot of focus on the CI side of things beyond the stuff that ArgoCD was doing, and Shipwright (which isn't really CI/CD focused but did some stuff around the actual build progress, but really suffered failure to launch).

PaulHoule 9mo ago

When Docker was new I had a really bad ADSL connection (2Mbps) and couldn't ever stack up a containerized system properly because Dockerhub would time out. I did large downloads all the time, I used to download 25GB games for my game consoles for instance. I just had to use schedule them and use tools that could resume downloads. If I'd had a local docker hub I might have used docker but because I didn't it was dead to me.

nyrikki 9mo ago

Unfortunately even podman etc.. are still limited by OCIs decision to copy the Docker model. Crun just stamp couples security profiles as an example, so everything in the shared kernel that is namespace incompatible is enabled. This is why it is trivial to get in-auditable communication between pods on a host etc…

ragall 9mo ago
> Unfortunately even podman etc.. are still limited by OCIs decision to copy the Docker model. Which parts of the model are you referring to ?
- nyrikki 9mo ago
  OCI Container Runtimes like OCI's runc are "container runtimes", so the runtime spec[2] Basically, docker started using lxc, but wanted a go native option, and wrote runc. If you look at [0] you can see how it actually instantiates the container. Here is a random blog that describes it fairly well [1] crun is the podman related project written in c, which is more efficient than the go based runc. You can try this even as the user nobody 65534:65534, but you may need to make some dirs, or set envs. Here is an example pulling an image with podman to make it easier, but you could just make an OCI spec bundle and run it: mkdir hello cd hello podman pull docker.io/hello-world podman export $(podman create hello-world) > hello-world.tar mkdir rootfs tar -C rootfs -xf hello-world.tar runc spec --rootless sed -i 's;"sh";"/hello";' config.json runc run container1 Hello from Docker! runc doesn't support any form of constraints like a bounding set on seccomp, selinux, apparmor, etc.. but it will apply profiles you pass it. Basically it fails open, and with the current state of apparmor and selinux it is trivial to bypass the minimal userns restrictions they place. Historically, before rootless containers this was less of an issue, because you had to be a privileged user to launch a container. But with the holes in the LSMs, no ability to set administrative bounding sets, and the reality that none of the defaults constrain risky kernel functionality like vsock, openat2 etc... there are a million ways to break netns isolation etc... Originally the docker project wanted to keep all the complexity of mutating LSM rules etc... in containerd. and they also fought even basic controls like letting an admin disable the `--privileged` flag at the daemon level. Unfortunately due to momentum, opinions, and friction in general, that means that now those container runtimes have no restrictions on callers, and cannot set reasonable defaults. Thus now we have to resort to teaching every person who launches a container to be perfect and disable everything, which they never do. If you run a k8s cluster with nodes on VMs, try this for example, if it doesn't error out, any pod can talk to any other pod on the node, with a protocol you aren't logging, and which has limited ability to log anyway. (if your k8s nodes are running systemd v256+ and you aren't using containerd which blocked vsock, but cri-o, podman, etc... don't (at least up to a couple of weeks ago) socat - VSOCK-LISTEN:3000 You can also play around with other af_families as IPX, Appletalk, etc... are all available by default, or see if you can use openat2 to use some file in /proc to break out. [0] https://manpages.debian.org/testing/runc/runc-spec.8.en.html https://manpages.debian.org/testing/runc/runc-spec.8.en.html [1] https://mkdev.me/posts/the-tool-that-really-runs-your-containers-deep-dive-into-runc-and-oci-specifications https://mkdev.me/posts/the-tool-that-really-runs-your-contai... [2] https://github.com/opencontainers/runtime-spec/blob/main/README.md https://github.com/opencontainers/runtime-spec/blob/main/REA...