3 ms·

I know it rarely happens in practice, but I disagree than Kubernetes needs an LTS since the Kubernetes cluster itself should be a "cattle not a pet" and thus, y

I know it rarely happens in practice, but I disagree than Kubernetes needs an LTS since the Kubernetes cluster itself should be a "cattle not a pet" and thus, you should "just" spin up a new cluster with the same version as your upgrade strategy.

dilyevsky 3y ago
Works great for clusters that dont have any storage, loadbalancers, any configuration drift…
simiones 3y ago
The whole point of Kubernetes is to move all of the machine management work to one place - the Kubernetes cluster nodes - so that your services are cattle, not pets. But there is no equivalent system for whole clusters. To transition from one cluster to another, you have to handle everything that k8s gives you without relying on k8s for it (restarting services, rerouting traffic, persistent storage etc). If you can do all that without k8s, why use k8s in the first place? So no, in practice clusters, just like any other stateful system, are not really cattle, they are pets you care for and carefully manage. Individual cluster nodes can come and go, but the whole cluster is pretty sacred.
- Kelkonosemmel 3y ago
  Global load balancer and traffic shifting is a real thing. But my preferred way is in place as it's easy: upgrade one Ctrl plane node after the other, than upgrade every node or create a new one and move your workload. The power of k8s is of course shit when you only move legacy garbage into the cloud as those are not ha at all. But lucky enough people slowly learn how to write apps cloud native.
  - simiones 3y ago
    Even with the best cloud native apps that can really be easily moved from one node to another, it is not easy to do cluster upgrades. Upgrading one control plane node after another can still break your cluster when you have your control plane nodes running incompatible versions. If you get a big worker node failover during the control plane upgrade, which of the control plane nodes should provision the new worker nodes, which versions of containers should kubelet pull, etc? The upgrade process for a running multi-node solution is an extremely difficult technical challenge, and it has to be designed very carefully to actually have a good chance of working. Breaking changes are almost impossible to introduce if the goal is to upgrade gracefully - you probably need a special compatible version to run while the upgrade is still in progress, that is neither the old version nor the targeted new version, but can handle both. You need to make sure all APIs are versioned in a way where it's easy to distinguish a request coming from an old client (so you can serve the old format) or a new one etc. Kubernetes has done some work on this, but it still often requires manual work. And many common plugins have just not put in this work and will break if you try to upgrade a live cluster.
    - Kelkonosemmel 3y ago
      Huh? I'm upgrading k8s for years now, had very rarely issues. Independent of it if you really have big workloads it's the right thing anyway to have a good team behind it. Nonetheless if you use gke for example I and no complicated plugins, it's basically automatic.
- edude03 3y ago
  > But there is no equivalent system for whole clusters. To transition from one cluster to another, you have to handle everything that k8s gives you without relying on k8s for it Ah, but there are. Maybe not a cluster orchestrator that would let you hot swap a single cluster, but if you're only running one cluster it's inherently a pet already. One common way that I've personally used to solve this problem is use Federated Clusters (kubefed) to run multiple clusters with global load balancing so that even when one entire cluster goes away its traffic is just shifted to the other cluster (whether because of upgrades or unexpected outages of the control plane)
  - simiones 3y ago
    Global load balancing is not enough, is it? If a request to save a new piece of info is served by one cluster, it still needs to update all the other clusters so that they can call retrieve it. So now you also need global data replication. And the data storage service shards need a name that's easily resolvable, so you need global name resolution across your clusters. And you need something to monitor the workloads so it can trigger failover, so you need global health checks too. And when you do an upgrade, you need to coordinate cordoning off the clusters etc - so you need global state as well. So overall, you need all, or most, of the features of kubernetes in your cluster federation solution. So why then not make each of these individual kubernetes clusters really small, say a single-node cluster each, and rely on the global infra to manage between them? Hey-presto, your "cattle farm" is now a single big pet.
oneplane 3y ago
That's what we do and it's great. It doesn't even apply to just Kubernetes, you should be able to swap out most components of your infrastructure without manual work, that's why we even invented all this modern software, or even hardware for that matter: redundant power supplies! If we can do that, why is everyone so scared of doing the same for other systems?

dilyevsky 3y ago

Works great for clusters that dont have any storage, loadbalancers, any configuration drift…

simiones 3y ago

The whole point of Kubernetes is to move all of the machine management work to one place - the Kubernetes cluster nodes - so that your services are cattle, not pets. But there is no equivalent system for whole clusters. To transition from one cluster to another, you have to handle everything that k8s gives you without relying on k8s for it (restarting services, rerouting traffic, persistent storage etc). If you can do all that without k8s, why use k8s in the first place? So no, in practice clusters, just like any other stateful system, are not really cattle, they are pets you care for and carefully manage. Individual cluster nodes can come and go, but the whole cluster is pretty sacred.

Kelkonosemmel 3y ago
Global load balancer and traffic shifting is a real thing. But my preferred way is in place as it's easy: upgrade one Ctrl plane node after the other, than upgrade every node or create a new one and move your workload. The power of k8s is of course shit when you only move legacy garbage into the cloud as those are not ha at all. But lucky enough people slowly learn how to write apps cloud native.
- simiones 3y ago
  Even with the best cloud native apps that can really be easily moved from one node to another, it is not easy to do cluster upgrades. Upgrading one control plane node after another can still break your cluster when you have your control plane nodes running incompatible versions. If you get a big worker node failover during the control plane upgrade, which of the control plane nodes should provision the new worker nodes, which versions of containers should kubelet pull, etc? The upgrade process for a running multi-node solution is an extremely difficult technical challenge, and it has to be designed very carefully to actually have a good chance of working. Breaking changes are almost impossible to introduce if the goal is to upgrade gracefully - you probably need a special compatible version to run while the upgrade is still in progress, that is neither the old version nor the targeted new version, but can handle both. You need to make sure all APIs are versioned in a way where it's easy to distinguish a request coming from an old client (so you can serve the old format) or a new one etc. Kubernetes has done some work on this, but it still often requires manual work. And many common plugins have just not put in this work and will break if you try to upgrade a live cluster.
  - Kelkonosemmel 3y ago
    Huh? I'm upgrading k8s for years now, had very rarely issues. Independent of it if you really have big workloads it's the right thing anyway to have a good team behind it. Nonetheless if you use gke for example I and no complicated plugins, it's basically automatic.
edude03 3y ago
> But there is no equivalent system for whole clusters. To transition from one cluster to another, you have to handle everything that k8s gives you without relying on k8s for it Ah, but there are. Maybe not a cluster orchestrator that would let you hot swap a single cluster, but if you're only running one cluster it's inherently a pet already. One common way that I've personally used to solve this problem is use Federated Clusters (kubefed) to run multiple clusters with global load balancing so that even when one entire cluster goes away its traffic is just shifted to the other cluster (whether because of upgrades or unexpected outages of the control plane)
- simiones 3y ago
  Global load balancing is not enough, is it? If a request to save a new piece of info is served by one cluster, it still needs to update all the other clusters so that they can call retrieve it. So now you also need global data replication. And the data storage service shards need a name that's easily resolvable, so you need global name resolution across your clusters. And you need something to monitor the workloads so it can trigger failover, so you need global health checks too. And when you do an upgrade, you need to coordinate cordoning off the clusters etc - so you need global state as well. So overall, you need all, or most, of the features of kubernetes in your cluster federation solution. So why then not make each of these individual kubernetes clusters really small, say a single-node cluster each, and rely on the global infra to manage between them? Hey-presto, your "cattle farm" is now a single big pet.

oneplane 3y ago

That's what we do and it's great. It doesn't even apply to just Kubernetes, you should be able to swap out most components of your infrastructure without manual work, that's why we even invented all this modern software, or even hardware for that matter: redundant power supplies! If we can do that, why is everyone so scared of doing the same for other systems?