6 ms·

Loosely, my understanding Kubernetes works like this: You have a Pod definition, which is basically a gang of containers. In that Pod definition you have inclu

Loosely, my understanding Kubernetes works like this: You have a Pod definition, which is basically a gang of containers. In that Pod definition you have included one or more container image references. You send the Pod definition to the API Server. The API Server informs listeners for Pod updates that there is a new Pod definition. One of these listeners is the scheduler, which decides which Node should get the Pod. It creates an update for the Pod's "status", essentially annotating it with the name of the Node that should run the Pod. Each Node has a "kubelet". These too subscribe to Pod definitions. When a change shows up saying "Pod 'foo' should run on Node 27", the kubelet in Node 27 perks up its ears. The kubelet converts the Pod definition into descriptions of containers -- image reference, RAM limits, which disks to attach etc. It then turns to its container runtime through the "Container Runtime Interface" (CRI). In the early days this was a Docker daemon. The container runtime now acts on the descriptions it got. Most notably, it will check to see if it has an image in its local cache; if it doesn't then it will try to pull that image from a registry. Now: The CRI is distinct from the Docker daemon API. The CRI is abstracted because since the Docker daemon days, other alternatives have emerged (and some have withered), such as rkt, podman and containerd. This update says "we are not going to maintain the Docker daemon option for CRI". You can use containerd. From a Kubernetes end-user perspective, nothing should change. From an operator perspective, all that happens is that you have a smaller footprint with less attack surface.

rsanheim 6y ago
I'm sure all this complexity makes sense for all sorts of reasons buried in the history of Kubernetes development at Google. "Things are the way they are because they got that way." The fact that so many other orgs, many of which are startups or just small to medium sized tech companies, use a system this complex is ludicrous to me.
- jacques_chester 6y ago
  Well, I might have designed it differently, but I wasn't there. For what it does, this architecture works well. More to the point: none of it is visible to end-users of Kubernetes. You send a Pod definition, some magic happens, pow! running software.
  - tannhaeuser 6y ago
    Except when it doesn't, at which point you're lost.
    - jacques_chester 6y ago
      Well, like I said, I might have done it differently. But there's a consistent logic to how it works[0]. That carries a lot of water. [0] most folks emphasise the control loop aspect, I think it's more helpful to point to blackboard / tuple-space systems as prior art.
      - tannhaeuser 6y ago
        > I think it's more helpful to point to blackboard / tuple-space systems as prior art Ah, nevermind then, that gives me rest concerning the viability and transparency of it all.
        jacques_chester 6y ago
        I'm glad I could help.
- comfydragon 6y ago
  The practical change that's happening is that in future versions of Kubernetes, they're removing support for using a shim for telling the Docker daemon to run containers, and focusing on just using containerd (which Docker uses under the covers anyway). It's kind of like if you had a shell script to launch programs, and it used to move the mouse to press icons, but now you've deprecated that and will only run programs directly.
- lima 6y ago
  Hi! Small tech company that uses k8s here. Much of the complexity is irreducible and has to go somewhere, and it's much better to have it in a single, stateless, well-defined place that's also easy too introspect. I've seen way too many Ansible nightmares grown out of deceptively "simple" mutable VM deployments. k8s makes our life so much easier because it eliminates a whole bunch of other complexity. Easily reproducible development environments, workload scheduling, sane config management...
  - LimaBearz 6y ago
    My favorite opposite of this is companies running k8s, using a ansible to build state on launch..
- mplewis 6y ago
  You don't need the complexity until you do, and then you're facing a rewrite into the complex system. Defining a deployment in YAML isn't really all that hard.
- yipbub 6y ago
  It's a generic API for automated provisioning. It asks you to make some decisions up front (what do you want to do when an instance goes down or has a new version available), and lets you forgo making some (like which nodes your instance should run on). It's not more complex. If you're small, the overhead is probably not worth it. If you're big enough, you manage the k8s control plane, and you don't have to manage your tenants infra. Is a programming language too complex for hello world? Perhaps, but that's not all it does.
freedomben 6y ago
You pretty much nailed it. This is a super useful "elevator description" to give to people. Mind if I share it (with attribution)? Even better if you slap it into a blog post or something (except a tweet thread :-D), but HN is perfectly fine :-)
- jacques_chester 6y ago
  Thankyou, I'm flattered. Please feel free to share.
chucky_z 6y ago
The GP did ask about logging though specifically. One of Docker daemon's more interesting features is how much log enrichment it does. Does kubelet do the same thing out of the box? I know containerd itself does not, unfortunately.
- jacques_chester 6y ago
  Yes, I overlooked that. I am afraid I don't know, but since the Docker daemon now relies on the same code, I would expect that there's similar functionality at that level.
kgoutham93 6y ago
Noob question: As I understand, dockershim makes docker daemon cri compliant. But dockerd already uses containerd which is cri compliant. So, why can't kubelet directly interact with containerd APIs without dockershim?
- jacques_chester 6y ago
  > So, why can't kubelet directly interact with containerd APIs without dockershim? Each kubelet does its thing through the Container Runtime Interface (CRI), so in a sense it doesn't know what it's running on. If it used containerd's interfaces directly, it wouldn't be possible to substitute in a different option. For example, there are emerging VM-based approaches like Firecracker and VMware "Project Pacific" (disclosure: I work at VMware).
- euank 6y ago
  The kubelet can talk to containerd's cri endpoint, yes, but there's one additional bit of complexity. If someone wants to use kubelet + docker so that they can, for example, ssh into a node and type 'docker ps' to see containers, or have something else using the docker api see the containers the kubelet started, that won't work after re-pointing the kubelet from docker to containerd. The difference here is namespacing[0], but not the linux-kernel-container-namespace, rather the containerd concept by the same name to allow "multi-tenancy" of a single containerd daemon. In addition, I don't think you could have docker + cri run in the same containerd namespace since they end up using different networking and storage containerd plugins. I think that terminology is right. So yeah, repointing the kubelet to containerd directly works fine, but it won't be the same thing as running docker containers. [0]: https://github.com/containerd/containerd/blob/9561d938/docs/namespaces.md#how-does-multi-tenancy-work https://github.com/containerd/containerd/blob/9561d938/docs/...

rsanheim 6y ago

I'm sure all this complexity makes sense for all sorts of reasons buried in the history of Kubernetes development at Google. "Things are the way they are because they got that way." The fact that so many other orgs, many of which are startups or just small to medium sized tech companies, use a system this complex is ludicrous to me.

jacques_chester 6y ago
Well, I might have designed it differently, but I wasn't there. For what it does, this architecture works well. More to the point: none of it is visible to end-users of Kubernetes. You send a Pod definition, some magic happens, pow! running software.
- tannhaeuser 6y ago
  Except when it doesn't, at which point you're lost.
  - jacques_chester 6y ago
    Well, like I said, I might have done it differently. But there's a consistent logic to how it works[0]. That carries a lot of water. [0] most folks emphasise the control loop aspect, I think it's more helpful to point to blackboard / tuple-space systems as prior art.
    - tannhaeuser 6y ago
      > I think it's more helpful to point to blackboard / tuple-space systems as prior art Ah, nevermind then, that gives me rest concerning the viability and transparency of it all.
      - jacques_chester 6y ago
        I'm glad I could help.
comfydragon 6y ago
The practical change that's happening is that in future versions of Kubernetes, they're removing support for using a shim for telling the Docker daemon to run containers, and focusing on just using containerd (which Docker uses under the covers anyway). It's kind of like if you had a shell script to launch programs, and it used to move the mouse to press icons, but now you've deprecated that and will only run programs directly.
lima 6y ago
Hi! Small tech company that uses k8s here. Much of the complexity is irreducible and has to go somewhere, and it's much better to have it in a single, stateless, well-defined place that's also easy too introspect. I've seen way too many Ansible nightmares grown out of deceptively "simple" mutable VM deployments. k8s makes our life so much easier because it eliminates a whole bunch of other complexity. Easily reproducible development environments, workload scheduling, sane config management...
- LimaBearz 6y ago
  My favorite opposite of this is companies running k8s, using a ansible to build state on launch..
mplewis 6y ago
You don't need the complexity until you do, and then you're facing a rewrite into the complex system. Defining a deployment in YAML isn't really all that hard.
yipbub 6y ago
It's a generic API for automated provisioning. It asks you to make some decisions up front (what do you want to do when an instance goes down or has a new version available), and lets you forgo making some (like which nodes your instance should run on). It's not more complex. If you're small, the overhead is probably not worth it. If you're big enough, you manage the k8s control plane, and you don't have to manage your tenants infra. Is a programming language too complex for hello world? Perhaps, but that's not all it does.

freedomben 6y ago

You pretty much nailed it. This is a super useful "elevator description" to give to people. Mind if I share it (with attribution)? Even better if you slap it into a blog post or something (except a tweet thread :-D), but HN is perfectly fine :-)

jacques_chester 6y ago
Thankyou, I'm flattered. Please feel free to share.

chucky_z 6y ago

The GP did ask about logging though specifically. One of Docker daemon's more interesting features is how much log enrichment it does. Does kubelet do the same thing out of the box? I know containerd itself does not, unfortunately.

jacques_chester 6y ago
Yes, I overlooked that. I am afraid I don't know, but since the Docker daemon now relies on the same code, I would expect that there's similar functionality at that level.

kgoutham93 6y ago

Noob question: As I understand, dockershim makes docker daemon cri compliant. But dockerd already uses containerd which is cri compliant. So, why can't kubelet directly interact with containerd APIs without dockershim?

jacques_chester 6y ago
> So, why can't kubelet directly interact with containerd APIs without dockershim? Each kubelet does its thing through the Container Runtime Interface (CRI), so in a sense it doesn't know what it's running on. If it used containerd's interfaces directly, it wouldn't be possible to substitute in a different option. For example, there are emerging VM-based approaches like Firecracker and VMware "Project Pacific" (disclosure: I work at VMware).
euank 6y ago
The kubelet can talk to containerd's cri endpoint, yes, but there's one additional bit of complexity. If someone wants to use kubelet + docker so that they can, for example, ssh into a node and type 'docker ps' to see containers, or have something else using the docker api see the containers the kubelet started, that won't work after re-pointing the kubelet from docker to containerd. The difference here is namespacing[0], but not the linux-kernel-container-namespace, rather the containerd concept by the same name to allow "multi-tenancy" of a single containerd daemon. In addition, I don't think you could have docker + cri run in the same containerd namespace since they end up using different networking and storage containerd plugins. I think that terminology is right. So yeah, repointing the kubelet to containerd directly works fine, but it won't be the same thing as running docker containers. [0]: https://github.com/containerd/containerd/blob/9561d938/docs/namespaces.md#how-does-multi-tenancy-work https://github.com/containerd/containerd/blob/9561d938/docs/...