I spent six years on Borg SRE, and two years so far working with Kubernetes, and this post reads like a strange combination of utopianism and adoration of obsol

I spent six years on Borg SRE, and two years so far working with Kubernetes, and this post reads like a strange combination of utopianism and adoration of obsolete ideas. Picking some parts to comment on at semi-random: > For that, let’s keep old versions of pod definitions > around, and make it trivial to “go back to version N”. > > [...] > > Bonus things you get from this: a diffable history of > what happened to your cluster, without needing GitOps > nonsense. By all means keep the GitOps nonsense if you > want, it has benefits, but you can answer a basic “what > changed?” question using only data in the cluster. This assumes either tiny clusters that will never run more than a hundred machines at a time, or an audit horizon measured in weeks. The clusters I help run today are much smaller than what some companies operate, but already we're hitting scaling issues with the sheer amount of data that sticks around during normal operation. If we had to store multiple copies of _pods_ for longer than a couple minutes then there wouldn't be an EBS volume with enough iops to handle background rescheduling. If you want an audit log, the obvious place is Git. I don't know why the OP derisively calls this "GitOps nonsense", because Google does the same thing and you should too. Figuring out what changed two months ago is much easier when each change has a commit message and a reviewer. > The latter is the bane of MetalLB’s existence, wherein it > gets into fights with other load-balancer implementations. > That should never happen. The orchestrator should have > rejected MetalLB’s addition to the cluster, because > LB-related fields would have two owners. To me the problem seems less an issue of field ownership, and more a problem of the network tier mutating parts of the workload scheduling tier. Why is MetalLB (or any other load balancer) changing Kubernetes state at all? Something has gone wrong here. The load balancer should watch the Kubernetes API to discover which endpoints exist and what their IPs are, and if it tries to _change_ state then that change should be blocked by the configured authorization policy. > So, for starters, let’s rip out all k8s networking. > Overlay networks, gone. Services, gone. CNI, gone. > kube-proxy, gone. Network addons, gone. If the author tries to start designing their new network stack they'll quickly have to put overlay networks and CNI and so on back in, because it turns out the real world gets a say in how we run our infrastructure and users need to be able to customize the boundary between Kubernetes and everything else. Kubernetes already suffers from insufficient customization in some areas, and networking is one of the few bright spots where it gives in and lets the operator do whatever we want to. IPv4? IPv6? Dual stack heterogenous routing? CNI lets you mix-n-match anything you can put into a binary as long as it can output JSON, and if it were even slightly more opinionated then it wouldn't be fit for purpose. > Let’s give every pod an IPv6 address. Yes, only an IPv6 > address for now. Where do they come from? Your LAN has a > /64 already (if it doesn’t, get with the program, I’m > not designing for the past here), so pluck IPs from there. > > [... lots of description of a thing that could be a > CNI plugin ...] > > That leaves bare metal clusters out in the cold, sort-of. > I argue this is a good thing, because there is no > one-size-fits-all load balancing. And this is why you absolutely don't want your workload scheduler to have opinions about networking. All of the stuff in there -- the hard requirement on an IPv6-aware fabric, the mission-impossible idea of routing traffic to pod IPs allocated at random from the full local subnet, the NAT64 (!!) -- can be done with a relatively small driver binary in any language that can call netlink, which means it's all possible to that in Kubernetes _today_ without foreclosing on the idea of running outside a carefully curated cloud environment. > We’re going to focus on doing one thing really well: if > you send me a packet for a pod, I’ll get the packet to > the pod. You can take that and build LBs based on LVS, > nginx, maglev-style things, cloud LBs, F5 boxes, the > world’s your oyster. And maybe I’ll even provide a couple > “default” implementations, as a treat. I do have lots of > opinions about load-balancers, so maybe I can make you a > good one. But the key is that the orchestration system > knows nothing about any of this, all it does is deliver > packets to pods. The workload scheduler (Kubernetes is a workload scheduler) shouldn't be in the business of delivering packets. That's up to the kernel and the network fabric. If your services' packets have to transit a userspace proxy on the way to their destination then you're already in trouble, and if that proxy is implemented by a second-order Wireguard overlay then all hope is lost. > I think this mostly translates to syncing more data down > to nodes in persistent storage, so that nodes have > everything they need to come back up into the programmed > state, even from a cold boot. Conceptually, I want the > cluster orchestrator to populate a set of systemd units > on each machine, and then switch to a very passive role > in the node’s life. > > [...] > > One way to view this is that in my “distributed” cluster > my pods are more likely to be unreplicated pets. It sounds like the author doesn't need Kubernetes at all. They want Puppet, or something Puppet-shaped, but with all the extra complexity that comes from having a distributed control plane. I don't know why they would want that, since if they're using Kubernetes at all then presumably they've got at least a few hundred machines being managed by a couple different product teams. There is a place for a central service that writes stuff to `/etc/` and assumes that individual machines are meaningful, but that service's target audience is completely separate from that of Kubernetes. There's no point in trying to design a replacement for Kubernetes to fit that market, any more than trying to design a 16-wheeler that competes with minivans.