3 ms·

How do they do version upgrades, isn't that the traditional Achilles heel of K8s that leads people to want to frequently recreate clusters from scratch and/or d

How do they do version upgrades, isn't that the traditional Achilles heel of K8s that leads people to want to frequently recreate clusters from scratch and/or do blue/green?

mfer 5y ago
First, if I understand it right... Google does some smart things in upgrades. They do things like tests and then upgrade their equivalent to AZs in a data center. I'm sure there have been upgrades gone bad that they've had to fix. Kubernetes can be upgraded. I've watched nicely done upgrades happening 4 or 5 years ago. I've watched simple upgrades happen more recently. It's not unheard of. Even in public clouds I've upgrade Kubernetes through many minor versions without issue. I would argue it's more work to create more clusters. You need to migrate workloads and anything pointing to them. It also would cost more as you have to run more hardware.
- deleted 5y ago
  [deleted]
- dharmab 5y ago
  We have active clusters that have been continually updated since 1.13 or so. Of course it is the cluster of Theseus since every single bit of compute has been entirely replaced :)
q3k 5y ago
If you don't depend on clusters being 100% available all the time and design your applications to handle cluster-wide outages (which you need to do at Google scale anyway), then simply doing progressive rollouts across clusters is good enough. If a cluster rollout fails and knocks some workloads offline, so be it. Just revert the rollout and figure out what went wrong. You can also throw in some smaller clusters that replicate the 'main' clusters' software stack and have some workloads whose downtime does not impact production jobs (not just dev workloads, but replicas of prod workloads!). These can be the earliest in your rollout pipeline and serve as an early stage canary.
jeffbee 5y ago
The borg concepts and interfaces have been the same for ages. The borglet and borgmaster get released and pushed out very frequently (daily or weekly) and it doesn't break anything among the workloads. There are not maintenance windows for these changes because containers can run while the borglet restarts, and borgmasters are protocol/format-compatible so the new release binary simply joins the quorum. Machines also get rotated out of the borg cell regularly for kernel upgrades, way more often than I've seen outside of Google. I think an important thing to know about K8s is Omega failed to replace Borg, and then the Omega people created K8s. So K8s does not necessarily descend from Borg, and not all of Borg's desirable attributes made it into K8s.
- dekhn 5y ago
  The omega folks were on the borgmaster and borglet teams when they were building omega (I was on the borg team at the same time, but working on a different project). It's fair to say that k8s is an intellectual inheritor of the parts of borg that are required for it to be useful on the outside. Borglet/borgmaster releases definitely break workloads. I recall one where something was rolled out to all the test clusters, passed all the tests (except one of ours) and was about to be promoted to increasing percents of the prod clusters. Whilst debugging why our test (which is not part of the feature rollout) broke we realize that if this had rolled out to prod, it would have broken all Tensorflow jobs, and would have been a major OMG. But yeah, most of the time, the release process for borglet and borgmaster is fairly fast and fairly reliable.
  - jeffbee 5y ago
    > required for it to be useful on the outside. A big part of this is that outside Google, the number of people who have to operate k8s as a fraction of all k8s users is way higher than the fraction of borg users who have to operate borg, so there's a lot of stuff in k8s that is 'end user experience' comforts and affordances for operators.
lokar 5y ago
Nothing important is single homed to one cluster. Much emphasis is placed on preventing correlated failed of clusters.
moondev 5y ago
Highly available kubeadm clusters are designed to be upgraded in place. The Kubernetes api-server is also designed to function with a minor version skew (for example v1.19.x and v1.20.x) that would happen during an in-place upgrade. Cluster API takes the above and can in-place upgrade clusters for you. It's pretty awesome to see first hand. Bumping the Kubernetes version string and machine image reference can upgrade and even swap distros of a running cluster with no down-time.
- dharmab 5y ago
  One caveat is that downgrading the apiserver is not guaranteed to be possible, since the schema of some types in the API may have been migrated to newer versions in the etcd database that the previous version may not be able to read. There are tools such as Velero (https://velero.io https://velero.io) which can restore a previous snapshot, but you will likely incur downtime and lose any changes since the snapshot.

How do they do version upgrades, isn't that the traditional Achilles heel of K8s that leads people to want to frequently recreate clusters from scratch and/or do blue/green?

mfer 5y ago

First, if I understand it right... Google does some smart things in upgrades. They do things like tests and then upgrade their equivalent to AZs in a data center. I'm sure there have been upgrades gone bad that they've had to fix. Kubernetes can be upgraded. I've watched nicely done upgrades happening 4 or 5 years ago. I've watched simple upgrades happen more recently. It's not unheard of. Even in public clouds I've upgrade Kubernetes through many minor versions without issue. I would argue it's more work to create more clusters. You need to migrate workloads and anything pointing to them. It also would cost more as you have to run more hardware.

deleted 5y ago
[deleted]
dharmab 5y ago
We have active clusters that have been continually updated since 1.13 or so. Of course it is the cluster of Theseus since every single bit of compute has been entirely replaced :)

q3k 5y ago

If you don't depend on clusters being 100% available all the time and design your applications to handle cluster-wide outages (which you need to do at Google scale anyway), then simply doing progressive rollouts across clusters is good enough. If a cluster rollout fails and knocks some workloads offline, so be it. Just revert the rollout and figure out what went wrong. You can also throw in some smaller clusters that replicate the 'main' clusters' software stack and have some workloads whose downtime does not impact production jobs (not just dev workloads, but replicas of prod workloads!). These can be the earliest in your rollout pipeline and serve as an early stage canary.

jeffbee 5y ago

The borg concepts and interfaces have been the same for ages. The borglet and borgmaster get released and pushed out very frequently (daily or weekly) and it doesn't break anything among the workloads. There are not maintenance windows for these changes because containers can run while the borglet restarts, and borgmasters are protocol/format-compatible so the new release binary simply joins the quorum. Machines also get rotated out of the borg cell regularly for kernel upgrades, way more often than I've seen outside of Google. I think an important thing to know about K8s is Omega failed to replace Borg, and then the Omega people created K8s. So K8s does not necessarily descend from Borg, and not all of Borg's desirable attributes made it into K8s.

dekhn 5y ago
The omega folks were on the borgmaster and borglet teams when they were building omega (I was on the borg team at the same time, but working on a different project). It's fair to say that k8s is an intellectual inheritor of the parts of borg that are required for it to be useful on the outside. Borglet/borgmaster releases definitely break workloads. I recall one where something was rolled out to all the test clusters, passed all the tests (except one of ours) and was about to be promoted to increasing percents of the prod clusters. Whilst debugging why our test (which is not part of the feature rollout) broke we realize that if this had rolled out to prod, it would have broken all Tensorflow jobs, and would have been a major OMG. But yeah, most of the time, the release process for borglet and borgmaster is fairly fast and fairly reliable.
- jeffbee 5y ago
  > required for it to be useful on the outside. A big part of this is that outside Google, the number of people who have to operate k8s as a fraction of all k8s users is way higher than the fraction of borg users who have to operate borg, so there's a lot of stuff in k8s that is 'end user experience' comforts and affordances for operators.

lokar 5y ago

Nothing important is single homed to one cluster. Much emphasis is placed on preventing correlated failed of clusters.

moondev 5y ago

Highly available kubeadm clusters are designed to be upgraded in place. The Kubernetes api-server is also designed to function with a minor version skew (for example v1.19.x and v1.20.x) that would happen during an in-place upgrade. Cluster API takes the above and can in-place upgrade clusters for you. It's pretty awesome to see first hand. Bumping the Kubernetes version string and machine image reference can upgrade and even swap distros of a running cluster with no down-time.

dharmab 5y ago
One caveat is that downgrading the apiserver is not guaranteed to be possible, since the schema of some types in the API may have been migrated to newer versions in the etcd database that the previous version may not be able to read. There are tools such as Velero (https://velero.io https://velero.io) which can restore a previous snapshot, but you will likely incur downtime and lose any changes since the snapshot.