4 ms·

This is a good start. It misses an important thing about Kubernetes that is often missed: extensibility. Each and every thing within Kubernetes can be swapped o

by hosh 2y ago

This is a good start. It misses an important thing about Kubernetes that is often missed: extensibility. Each and every thing within Kubernetes can be swapped out for something else, including the scheduler. There are Custom Resource Definitions that supports these extensions and operators. For example, there is no built-in autoscaler for nodes, but someone wrote one and you can add one in there. It uses a constraint solver for determining whether to expand or shrink node groups. If you want to use something else, you can find something or write it and install it. Another example, you don't have to use kube-proxy. There are other ways to manage inter-node networking. To address some of the questions: > Why does it not matter if the state is unstable? If I'm operating a cluster that can't settle, I'd like to know immediately! I'd point to the Cynefine framework to help make sense of this. Kubernetes is a tool that helps manage things in the complex domain, rather than the complicated domain. Unlike complicated systems, complex systems and complex adaptive systems may never reach a defined settled state. > Basically, Horizontal Pod Autoscaler but with sensors which are not just "CPU" That's already available. In addition to scaling on built-in metrics such as cpu and mem, there are ways to create custom metrics, including queue depth. You can do this because Kubernetes is extensible > Why are the storage and networking implementations "out of tree" (CNI / CSI)? It used to be in-tree, until the number of third party cloud and in-house storage and networking providers became unwieldy. This goes along with that fundemental characteristic of the Kubernetes design -- extensibility. AWS owns the Elastic Block Storage CSI, and GCP owns its CSI for its storage device. CNI allowed for the various service meshes, including exciting new ones such as the one based on eBFP. The Cynefine framework again, helps sheds some light on this: the best way to respond to things in the complex domain is to try a lot of different approaches. Even if one approach doesn't look like it works now, some future state may make that previously impractical approach to work well. > Given the above question, why is there explicit support for Cloud providers? The current versions of Kubernetes pushes those implementations to CNI and operators. So for example, in order to make use of AWS ELB/ALB for the Ingress object, you have to additionally install the AWS-specific driver. If you are using AWS's EKS service, this is managed as an EKS addon. Under the hood, these drivers are, guess what, pods managed by replicasets managed by deployments that listens to the Kubernetes API servers for changes to the Ingress resource. Not everyone uses it. On one of the Kubernetes sites I worked on, we used Traefik inside Kubernetes and its custom resource definition, IngressRoute. Everytime you create an Ingress, the AWS driver will create a completely new Load Balancer, which of course, drives up cost for very little gain.

deivid 2y ago
Thanks for the detailed answer > extensibility This is something that irks me right now, but at my current knowledge level (~0) it didn't feel right to even mention: If Kubernetes pushes everything out (CSI, CNI, Cloud provider integration, LoadBalancer, ...), doesn't it become "just a control loop runner"? I'm sure there's value in that, but I can imagine that most people running clusters would make different choices for various options, landing you with _every cluster being unique_, which feels wrong somehow.
- llarsson 2y ago
  That's right, and it's also why you see Kubernetes distributions popping up. That way, someone has already done all the configuration of various plugins and components for you. For instance, the major clouds that all let you easily start with their Kubernetes services and that they integrate well with the logging and monitoring systems, IAM, server/node provisioning, etc. Or ones that are not tied to any particular cloud provider, and perhaps focused on security (full disclosure: I work for Elastisys, who makes exactly that) or ease of use. I'm sure we will see more and more such efforts in this space, exactly because cobbling together something yourself from scratch (basically just a control loop runner, as you said) is neither very appealing when you think about ongoing maintenance, nor very cost-effective for businesses to spend engineering time on.
- hosh 2y ago
  That is correct. Every cluster is unique. What we have instead are a set of standard design patterns that are broadly (but not universally) applicable. Even the core components can be swapped out so that Kubernetes can be adapted. For example, there are several teams that wrote kubelet replacements. The built-in scheduler for kubernetes is only responsible for pod placement by contacting the kubelet on a node and changing its desired state. The kubelet is responsible for actual placement. That meant people have created kubelets that could run wasm workloads as pods. This was how native Windows workloads were implemented. I remember correctly, there are also kublets that could run unikernels, microvms, etc. The design pattern of label selectors remains the same, so that same concept can be used to place those specialized workloads. I also learned something from an interview I once had ... their platform team came to the conclusion that every site is unique because every company will have its own specific needs and unique combination of technology. They (deliberately) don't use Kubernetes. They even crafted their interview process for that. It's a timed session to attempt to do something deceptively simple, that turned out to be an ops nightmare. There are no internet resources describing how to do that. No one is expected to finish it within the time. Instead, they are looking at how someone work through the problem as well as their emotional regulation while under time pressure. Finally, I'd like to draw attention to a fallacy that has plauged modernity, and it is very applicable to computing design. This is from the field of political science, from a book written by James C. Scott called "Seeing Like a State". There is a blog post about the key concept -- legibility -- https://www.ribbonfarm.com/2010/07/26/a-big-little-idea-called-legibility/ https://www.ribbonfarm.com/2010/07/26/a-big-little-idea-call... The pursuit of legibility has lead us to imposing a overly simplistic view onto something complex instead of really understanding the complexity. In the case of governments, this imposition uses authoritarian powers, and results of fragile solutions. In the case of computing infrastructure, we create abstractions and are surprised when it leaks.

by hosh 2y ago

deivid 2y ago

Thanks for the detailed answer > extensibility This is something that irks me right now, but at my current knowledge level (~0) it didn't feel right to even mention: If Kubernetes pushes everything out (CSI, CNI, Cloud provider integration, LoadBalancer, ...), doesn't it become "just a control loop runner"? I'm sure there's value in that, but I can imagine that most people running clusters would make different choices for various options, landing you with _every cluster being unique_, which feels wrong somehow.

llarsson 2y ago
That's right, and it's also why you see Kubernetes distributions popping up. That way, someone has already done all the configuration of various plugins and components for you. For instance, the major clouds that all let you easily start with their Kubernetes services and that they integrate well with the logging and monitoring systems, IAM, server/node provisioning, etc. Or ones that are not tied to any particular cloud provider, and perhaps focused on security (full disclosure: I work for Elastisys, who makes exactly that) or ease of use. I'm sure we will see more and more such efforts in this space, exactly because cobbling together something yourself from scratch (basically just a control loop runner, as you said) is neither very appealing when you think about ongoing maintenance, nor very cost-effective for businesses to spend engineering time on.
hosh 2y ago
That is correct. Every cluster is unique. What we have instead are a set of standard design patterns that are broadly (but not universally) applicable. Even the core components can be swapped out so that Kubernetes can be adapted. For example, there are several teams that wrote kubelet replacements. The built-in scheduler for kubernetes is only responsible for pod placement by contacting the kubelet on a node and changing its desired state. The kubelet is responsible for actual placement. That meant people have created kubelets that could run wasm workloads as pods. This was how native Windows workloads were implemented. I remember correctly, there are also kublets that could run unikernels, microvms, etc. The design pattern of label selectors remains the same, so that same concept can be used to place those specialized workloads. I also learned something from an interview I once had ... their platform team came to the conclusion that every site is unique because every company will have its own specific needs and unique combination of technology. They (deliberately) don't use Kubernetes. They even crafted their interview process for that. It's a timed session to attempt to do something deceptively simple, that turned out to be an ops nightmare. There are no internet resources describing how to do that. No one is expected to finish it within the time. Instead, they are looking at how someone work through the problem as well as their emotional regulation while under time pressure. Finally, I'd like to draw attention to a fallacy that has plauged modernity, and it is very applicable to computing design. This is from the field of political science, from a book written by James C. Scott called "Seeing Like a State". There is a blog post about the key concept -- legibility -- https://www.ribbonfarm.com/2010/07/26/a-big-little-idea-called-legibility/ https://www.ribbonfarm.com/2010/07/26/a-big-little-idea-call... The pursuit of legibility has lead us to imposing a overly simplistic view onto something complex instead of really understanding the complexity. In the case of governments, this imposition uses authoritarian powers, and results of fragile solutions. In the case of computing infrastructure, we create abstractions and are surprised when it leaks.