7 ms·
A skeptic's first contact with Kubernetes
- deleted 2y ago[deleted]
- jauntywundrkind 2y ago> Why are the storage and networking implementations "out of tree" (CNI / CSI)? Given the above question, why is there explicit support for Cloud providers? eg: LoadBalancer supports AWS/GCP/Azure/.. Kubernetes has been pruning out vendor-specific code for a while now, moving it out of tree. The upcoming 1.31 release will drop a lot of existing, already deprecated support for AWS & others from Kubernetes proper. https://github.com/kubernetes/enhancements/blob/master/keps/sig-cloud-provider/2395-removing-in-tree-cloud-providers/README.md https://github.com/kubernetes/enhancements/blob/master/keps/... There's some plan to make this non-dosruptive to users but I haven followed it closely (I don't use these providers anyhow). > Why are we generating a structured language (YAML), with a computer, by manually adding spaces to make the syntax valid? There should be no intermediate text-template representation like this one. Helm is indeed a wild world. It's also worth noting that Kubernetes is also pushing towards neutrality here; Helm has never been an official tool, but Kustomzie is builtin to kubectl & is being removed. https://github.com/orgs/kubernetes/projects/183/views/1?filterQuery=+sig%3A%22sig-cli%22&pane=issue&itemId=66600030 https://github.com/orgs/kubernetes/projects/183/views/1?filt... There's a variety of smart awesome options out there. First place I worked at that went to kube used jsonnet (which alas went unmaintained). Folks love CUE and Dhall and others. But to my knowledge there's no massive bases of packaged software like exists for Helm. Two examples, https://github.com/bitnami/charts/tree/main/bitnami https://github.com/bitnami/charts/tree/main/bitnami https://github.com/onedr0p/home-ops https://github.com/onedr0p/home-ops . It'd be lovely to see more works outside Helm. Thanks sysdig for your 1.31 write up, https://sysdig.com/blog/whats-new-kubernetes-1-31/ https://sysdig.com/blog/whats-new-kubernetes-1-31/
- deivid 2y ago> Kubernetes has been pruning out vendor-specific code for a while now, moving it out of tree. I wasn't aware, but it makes sense, and explains the current "confusing" state.
- jauntywundrkind 2y agoThanks for the write-up, and especially for storytelling your ability & willingness to wade in & find out! Proper hacker spirit! You did such a great job offering an informative high level view (honing in on the control loop feels like seizing upon a critical insight), & iterating through bits nicely. You were tactful & clear in raising some ongoing qualms, which indeed seem largely correct & frustrating, even to those who started as believers. Thanks thanks, a recommendable article indeed.
- bigstrat2003 2y agoOff topic - thank you for not having a super narrow text layout for your site. It seems like every other website these days has an incredibly narrow text width (I've seen as small as 600px which is so annoying to read). It was like a breath of fresh air to go to your site and have the text be a reasonable width without me having to fiddle with page styles.
- mtndew4brkfst 2y agoKustomize is builtin to kubectl & is being removed Hadn't heard this until now, I'm a rather happy user after being jaded about Helm 2/3. Do you happen to know if this removal is sentimentally closer to "we don't want to keep maintaining kustomize" or to "we don't want to keep embedding kustomize, please use its binary directly"?
- liggitt 2y agoThe motivation is more the latter, but it's not at all clear the proposed removal of the embedded kustomize will proceed, given the compatibility implications. See discussion at https://github.com/kubernetes/enhancements/issues/4706#issuecomment-2165806646 https://github.com/kubernetes/enhancements/issues/4706#issue... and following.
- mtndew4brkfst 2y agoThank you for this and your many many other contributions!
- JojoFatsani 2y agoThe helm yaml thing really is annoying. Unfortunately it feels like helm is too firmly embedded to unseat at this point.
- akvadrako 2y agoYou don't need helm at all. I've used k8s at a few places, big and small, and we just used kustomize or simple scripts to generate the actualized configs.
- IceWreck 2y agoI've switched to kustomize for my homelab. Its built into kubectl (no extra tool needed) and its not really templating. It doesn't fully replace helm for 3rd party tools or packaging but its a solid alternative for less complicated setups.
- worldsayshi 2y agoFunny then that kustomize apparently is being removed, according to another comment: https://news.ycombinator.com/item?id=41093797 https://news.ycombinator.com/item?id=41093797
- dv_dt 2y agoMy annoyance with helm is that it's possible for a helm install to fail, with no top level install status, yet have failed chart resources where you need to go digging around and manually delete before another install can be applied. I mean helm has one key job..
- azaras 2y agoI agree with: > My opinion is that a large part of Kubernetes' value is derived from just two concepts I agree with the first one, "control loops," but the second one is "API-based," not "Services."
- hbogert 2y agoHis take on text interpolation is very right. I'm a SWE turned SRE because as a developer I really enjoyed using K8s. But as a full-time SRE where I work just means YAML juggling. It's mind numbing that everybody is okay with this, this really is our domain's assembly era, albeit with whitespace, colons, dashes and brackets. I've found solace in CUE which I just run locally to catch all the small errors everybody makes on a daily basis. Putting the CUE validation in our pipeline is too confronting for others, yet they're constantly making up best practices adhoc during reviews which could've easily been codified with CUE (or some other serious config language).
- jauntywundrkind 2y agoThere's a ton of discussion of alternative ideas as comments. Many look amazing. But I keep wondering how many would be useful in giving us the modularity to make something like the shareable charts Helm has. That bitnami and others have these massive troves of charts is such a superpower for Kubernetes. We all seem to agree it's not the string templating that's excellent, but figuring out how to package & make modular the templates/generators, and how to manage the outputs of the generators: that's "the rest of the owl" that tools such as Kustomzie and languages such as Dhall and starlark don't really buy us. There is MetaController, which seems like some way to operationalize running generators and managing outputs. But I'm not sure what good examples there are of its use and specifically what examples are intended for distribution alike Helm Charts sometimes are.
- galdosdi 2y ago> Putting the CUE validation in our pipeline is too confronting for others Sad. Can you get away with boiling the frog by getting it initially added but configured to check very, very little? Maybe some specific rare category of mistake that doesn't happen enough to annoy your colleagues when it catches it but is implicated in a recent outage management still remembers? Then slowly adding rules over time as feasible (exploiting each outage as an opportunity to enlist stakeholder support for adding more rules that would have caught that particular misconfiguration event) Sometimes I think figuring out how to stage and phase gradually making the changes you need to improve the system in light of social inertia against it is the most complex part of corporate enterprise software work. I definitely remember a time I wrote a whole perl-based wheel-reinventing crappy puppet knockoff just for a set of nagios plugins, entirely not for technical reasons, but for the political reason that this way we would control it instead of the other department which refused to get with the program and let us do staged canary rollouts. It was wrong technically if you assume frictionless corporate politics, but it was the right and only practical way to achieve the goal of ending the drumbeat of customer outages.
- mcorbin 2y agoNice article ;) > The number of rules may be large, which may be problematic for certain traffic forwarding implementations (iptables linearly evaluates every rule) Kube proxy also supports ipvs out of the box, and some CNI (like Cilium) can also replace kube proxy and rely on eBPF. > When a Pod is moved to another node, the traffic will be forwarded twice until the old DNS entry expires Not sure to understand this one. On a standard setup what happens is: - Pod A is running on a node, receiving traffic from a service - The pod is stopped by kubelet (that send a SIGTERM to it). - The pod should gracefully shutdown. During the shutdown phase, only _existing_ connections are forwarded to the stopping pod, new ones will be already forwarded elsewhere. - If the pod stops before the terminationGracePeriodSeconds duration (default 30s), everything is fine. Else, the pod is killed by kubelet. So it's developers that should make sure pods handle signals correctly. "Misbehaving clients (eg: ones that do not re-resolve DNS before reconnecting) will continue to work" => the services IP is stable so clients don't need to re-resolve. > Why does it not matter if the state is unstable? If I'm operating a cluster that can't settle, I'd like to know immediately! Kubernetes exposes a lot of metrics, on the control plane components or kubelet, usually using the Prometheus format. Look for example at the metrics exposed by kube state metrics: https://github.com/kubernetes/kube-state-metrics/tree/main/docs/metrics https://github.com/kubernetes/kube-state-metrics/tree/main/d... With controllers metrics + kube state metrics about most Kubernetes resources, you can easily build alerts when a resource fails to reconcile. > Basically, Horizontal Pod Autoscaler but with sensors which are not just "CPU" Take a look at KEDA, it's exactly this: https://keda.sh/ https://keda.sh/ It "extends" the autoscaler capabilities. If you're running Prometheus you can for example scale on any metric that is stored in Prometheus (and so exposed by your application/infrastructure components: queue depth, latency, request rate...). Kubernetes was built to be extended like this. Same for your question "Why are the storage and networking implementations "out of tree" (CNI / CSI)?", to my experience support is very good today on various cloud providers or on premise infra components. Look at Karpenter for example, it's IMO a revolution in the Kubernetes node management world.
- politelemon 2y agoThis was a useful read and somewhat gels with my experiences. Looking at the statement at the beginning > and it only requires you to package your workload as a Docker image, which seems like a reasonable price to pay. is no longer true as you continue down the path, since it actually requires you to do a lot more than you'd think.
- deivid 2y agoI've not yet gone down that path, in which other ways does your workload need to adapt? I understand that you may want to get more value out of logs, and include side-cars, you may want inter-pod encryption and include service meshes, etc; but that'd be something that requires extra complexity in any setup
- movedx 2y agoOIDC integration, RBAC, data persistence/replication across nodes, observability, mTLS. And yes, you're right, all these things are complex in any situation, except when you simply use a load balancer and two servers. There are companies estimated to be worth close to US$2B using with sub 30 servers using a service you may have heard of: Stack Overflow (https://nickcraver.com/blog/2016/03/29/stack-overflow-the-hardware-2016-edition/ https://nickcraver.com/blog/2016/03/29/stack-overflow-the-ha... && https://nickcraver.com/blog/2016/05/03/stack-overflow-how-we-do-deployment-2016-edition/ https://nickcraver.com/blog/2016/05/03/stack-overflow-how-we...). (As an aside: K8s does not free you from your Cloud vendor - after floating around my city working on various "platforms", all locked in due to IAM, for example.)
- xyzzy_plugh 2y agoNone of those things are specific to Kubernetes, though. If anything it's a great forcing function to do the things you aught to for non-k8s deployments. It's far easier to write, say, a systems unit for a service that is properly configurable, has health checks, etc.
- lmm 2y ago
- cyberax 2y agoMy problem with K8s: the network abstraction layer just feels _wrong_. It's an attempt to replicate the old model of "hard exterior, gooey interior" model of corporate networks. I would very much prefer if K8s used public routable IPv6 for traffic delivery, and then simply provided an authenticated overlay on top of it.
- icedchai 2y agoI've felt similarly. Possibly because I was online pretty early, pre-NAT... there was public IPv4 everywhere.
- kasey_junk 2y agoSomething like? https://john-millikin.com/stateless-kubernetes-overlay-networks-with-ipv6 https://john-millikin.com/stateless-kubernetes-overlay-netwo...
- cyberax 2y agoIt's still an overlay network.
- dilyevsky 2y agoHow do you suggest talking to ipv4-only internet hosts and supporting ipv4-only containers?
- cyberax 2y agoVia the border load balancers, just like we do it now.
- dilyevsky 2y agoSo basically i need to run another piece of infra that does NAT64 and DNS64 and limits my deployment options quite a bit (can't do DSR)? Totally unnecessary in cloud... Not sure how that's better for users but probably better for vendors ;) Btw, overlay is not the only option to do CNI - Calico, Cilium and few others can do it via l3 by integrating with your equipment. Even possible in cloud but has serious scale limitations...
- INTPenis 2y agoReminds me of myself, and probably many others on here. I've always been a skeptic of novelty. Just like with cryptocoins I was there on the ground floor, I remember when Docker was launching. I remember trying to understand what it was, and I remember someone comparing it to MSI for Linux. Unlike cryptocoins I didn't miss out on being a millionaire with containers. :( I just avoided containers until 2019! So to me it was first containers, and then kubernetes. That way I was already sold on the whole container image concept and namespaces in your OS, I had used FreeBSD jails in the early 2000s. So when I understood k8s I realized it's literally just a container orchestrator. It might seem complicated but that's all to do with being able to run containers on a range of nodes instead of just one. And of course having an API to tie it all together. Whether you project needs that or not is something you should definitely explore in depth before you set out and try to use it. Personally I prefer container hosts over k8s for most startup projects. I look forward to Talos' new podman package and being able to deploy a minimal podman container host with Talos, no k8s necessary.
- nrr 2y ago"Whether you project needs that or not is something you should definitely explore in depth before you set out and try to use it." Unfortunately, as someone who's written these kinds of orchestrators a few times already (going back to jails on FreeBSD 4.8 and Xen domUs on NetBSD 2.0) and has a nose for where they're a good idea, I find this is overwhelmingly not the prevailing wisdom. I desperately wish it were. Somewhere, the industry went all-in on Kubernetes, and I have zero desire to touch most of the resulting software stacks. We sold folks hard on complexity being necessary, and Kubernetes makes it very tempting to go ham with it.
- Atreiden 2y agoGreat writeup on the core fundamentals, saved this to share with engineers who are new to k8s and need a quick primer. Re: This piece - > Given the Controller pattern, why isn't there support for "Cloud Native" architectures? > I would like to have a ReplicaSet which scales the replicas based on some simple calculation for queue depth (eg: queue depth / 16 = # replicas) > Defining interfaces for these types of events (queue depth, open connections, response latency) would be great > Basically, Horizontal Pod Autoscaler but with sensors which are not just "CPU" HPAs are actually still what you want here - you can configure HPAs to scale automatically based on custom metrics. If you run Prometheus (or a similar collector), you can define the metric you want (e.g. queue-depth) and the autoscaler will make scaling decisions with these in mind. Resources: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#scaling-on-custom-metrics https://kubernetes.io/docs/tasks/run-application/horizontal-... https://learnk8s.io/autoscaling-apps-kubernetes https://learnk8s.io/autoscaling-apps-kubernetes
- entropyie 2y agoSee KEDA and Karpenter for advanced k8s acaling
- zikduruqe 2y ago^ this right here. We used KEDA to query DynamoDB to look at a queue depth we wrote to a table. If number was X, then we would scale on it. Was pretty slick.
- kryptn 2y agoKarpenter is great for managing spot fleet nodes as well. Most of our clusters run a small aws managed node group for karpenter and the rest of the nodes will be spot fleet and managed by karpenter.
- deivid 2y agoUsing custom/external metrics is exactly what I was looking for, thanks! I think my misunderstanding comes from the distinction between "first party" sensors (CPU scaler) and "third party" sensors (via "external metrics"). Is there a reason for this distinction? Will the CPU scaler eventually be pushed out of k8s as well?
- hosh 2y agoThis is a good start. It misses an important thing about Kubernetes that is often missed: extensibility. Each and every thing within Kubernetes can be swapped out for something else, including the scheduler. There are Custom Resource Definitions that supports these extensions and operators. For example, there is no built-in autoscaler for nodes, but someone wrote one and you can add one in there. It uses a constraint solver for determining whether to expand or shrink node groups. If you want to use something else, you can find something or write it and install it. Another example, you don't have to use kube-proxy. There are other ways to manage inter-node networking. To address some of the questions: > Why does it not matter if the state is unstable? If I'm operating a cluster that can't settle, I'd like to know immediately! I'd point to the Cynefine framework to help make sense of this. Kubernetes is a tool that helps manage things in the complex domain, rather than the complicated domain. Unlike complicated systems, complex systems and complex adaptive systems may never reach a defined settled state. > Basically, Horizontal Pod Autoscaler but with sensors which are not just "CPU" That's already available. In addition to scaling on built-in metrics such as cpu and mem, there are ways to create custom metrics, including queue depth. You can do this because Kubernetes is extensible > Why are the storage and networking implementations "out of tree" (CNI / CSI)? It used to be in-tree, until the number of third party cloud and in-house storage and networking providers became unwieldy. This goes along with that fundemental characteristic of the Kubernetes design -- extensibility. AWS owns the Elastic Block Storage CSI, and GCP owns its CSI for its storage device. CNI allowed for the various service meshes, including exciting new ones such as the one based on eBFP. The Cynefine framework again, helps sheds some light on this: the best way to respond to things in the complex domain is to try a lot of different approaches. Even if one approach doesn't look like it works now, some future state may make that previously impractical approach to work well. > Given the above question, why is there explicit support for Cloud providers? The current versions of Kubernetes pushes those implementations to CNI and operators. So for example, in order to make use of AWS ELB/ALB for the Ingress object, you have to additionally install the AWS-specific driver. If you are using AWS's EKS service, this is managed as an EKS addon. Under the hood, these drivers are, guess what, pods managed by replicasets managed by deployments that listens to the Kubernetes API servers for changes to the Ingress resource. Not everyone uses it. On one of the Kubernetes sites I worked on, we used Traefik inside Kubernetes and its custom resource definition, IngressRoute. Everytime you create an Ingress, the AWS driver will create a completely new Load Balancer, which of course, drives up cost for very little gain.
- skywhopper 2y agoWhy is he mad at Kubernetes about Helm? Yes, Helm is a mess. Yes, lots of people use Helm to deploy things to Kubernetes, but critiques of Helm are not valid critiques of Kubernetes.
- andix 2y agoAs long as there is no popular replacement for Helm, it will stay tightly coupled with Kubernetes. Not in a technical way, but in the way that you're always going to use it together with k8s. For now Helm is the only sane way to package a complex k8s deployment in a reusable way. Kustomize is great too, but quite limited in many ways.
- sigwinch28 2y agoIf we replace “YAML” with “JSON” and then talk about naive text-based templating, it seems wild. That’s because it is. Then we go back to YAML and add whitespace sensitivity and suddenly it’s the state-of-the-art for declaring infrastructure.
- mark242 2y agoHere's what I would like from a Kubernetes-like system. I have a collection of machines. I have a very simple file that defines A) what I want to run, B) how the pieces communicate to each other, and C) how I want it to scale. Make that work without me having to think about any of the underlying infrastructure. (If this sounds suspiciously similar to Heroku, there you go)
- jfengel 2y agoI think I'd be even happier if it had reasonable defaults for C. At least as a novice, I probably don't know how it wants to scale, and should be able to figure it out better than me.
- hosh 2y agoThere are tools that can do that, and are not as complex as Kubernetes I once interviewed at a place where their notion of "scaling" is to turn up the knob on Heroku. They also pay for it. They did not have anyone on their team who knew what questions to ask. They had grown to the point where they had outgrown their approach, and yet, have never developed the in-house capability. I mentioned the Cynefine framework elsewhere, and I'll mention it again. Infrastructure and how it supports application, users, stakeholders, is a complex system. A lot of people treat this as if it were in the complicated, or even clear domain. For certain class of use-cases, you can get away with this. But a company itself is a complex adaptive system, whose needs changes as it matures and scales ... or it doesn't, and it collapses. The Cynefine framework describes four different domains, and the way you make decisions are different for each of them. You start getting in trouble when you attempt to use the decision-making process for one domain to a different domain. This often happens when the situation has evolved without the decision-makers noticing. It is easier for experts to see how clear domains changes to complicated domains, but it is more difficult to see when complicated domains change to complex domains. Kubernetes is a sophisticated, adaptive, extensible tool for the complex domain. If it seems overly complex, it is because the domain is complex, and is not necessary if you are working from the complicated or clear domain.
- movedx 2y ago
- AcerbicZero 2y agoThis was a solid write up; I've been using K8s (intermittently) for like, 5 years now, and I still spend an inordinate amount of time looking things up and trying to convert the nonsense naming conventions used to something understandable. I can think of 20 or so projects that would have run great on K8s, and I can think of 0 projects that were running on K8s, which worked well. Eventually, seeing the wrong tool used for the wrong job time and time again I came around to seeing K8s as the latest iteration of time sharing on a mainframe, but this time with YAML, and lots of extra steps.
- cybrexalpha 2y ago> Why are we generating a structured language (YAML), with a computer, by manually adding spaces to make the syntax valid? Yep, it sucks. It's not like nobody has tried to do better, but nothing else has the adoption of Helm. Ultimately text is, as always, universal. If you want a fun fact: the communication between kubectl and the kube-api-server is actually in JSON, not YAML.
- movedx 2y ago> If you want a fun fact: the communication between kubectl and the kube-api-server is actually in JSON, not YAML. YAML is a super set of JSON. All JSON is valid YAML. No one uses YAML over the wire as you cannot guarantee the conversion from YAML to JSON as YAML is a super set and make contain things like anchors, which the JSON parser cannot handle. That's why :)
- zbentley 2y agoNobody tell the JSON::XS CPAN maintainer you said that: https://metacpan.org/pod/JSON::XS#JSON-and-YAML https://metacpan.org/pod/JSON::XS#JSON-and-YAML Or John Millikin: https://john-millikin.com/json-is-not-a-yaml-subset https://john-millikin.com/json-is-not-a-yaml-subset
- mst 2y agoOne of many reasons why most of the perl devs I know use the Cpanel::JSON::XS fork now.
- FridgeSeal 2y agoSolid write up, but small nitpick with the diagram at the start: It displays a pod containing multiple containers (this is fine and normal) but then highlights some of those containers to be different services. Unless you guys are running wildly different setups, or we’re talking about sidecar and init containers which is a whole rabbit hole unto itself, I put different services in different pods.
- globular-toast 2y agoAlso the same diagram shows nodes being lower in the hierarchy than namespaces. Namespaces and nodes are orthogonal.
- jongjong 2y agoKubernetes provides a relatively simple abstraction to represent a cluster of machines as a single entity. It's just about as simple as it can be IMO. Its simplicity leverages the idea that each part of your software should be fully responsible for its own lifecycle and can handle and recover from all scenarios that can impact it. For example, if your app service happens to launch before your database service, then it should be able to seamlessly handle this situation and keep trying to reconnect until the database service is started. This characteristic is generally desirable in any operating environment but it is critical for Kubernetes. Its orchestration model is declarative, so it doesn't make much fuss over launch order... Yet it works well and simply if each of your services have this failure tolerance and self-healing characteristic.
- actionfromafar 2y agoWny can't we use Meson? Yaml is basically used as a bizarre build system anyways.
- jgalt212 2y agoWhat's the break-even number of machines you must manage before Kubernetes starts to make sense?
- wavemode 2y agoKubernetes starts to make sense when you don't know the number of machines you need (and therefore desire auto-scaling based on system load).
- lmm 2y agoI'd say 3 or more distinct technology stacks. You can run 200-machine clusters comfortably with lighter tooling if they're all Ruby or all JVM or what have you (and I have), but once your deployment process requires remembering the foibles of 4 different Python build tools it becomes easier to shove it all in Kubernetes.
- JoosToopit 2y ago[flagged]
- dilyevsky 2y agoOn the last point ("Stringy Types") - k8s API types are actually defined as Protobufs[0] so they have strictly defined schemas. There are some types that are sum types (IntOrString) but generally no type depends on some other field's value afaik. Ofc that doesn't stop CRD developers from making everything a String type and then interpolating server-side (within their custom controller) based on phases of the moon and what not... [0] - e.g https://github.com/kubernetes/api/blob/master/core/v1/generated.proto https://github.com/kubernetes/api/blob/master/core/v1/genera...