10 ms·

If I've read the Google papers on borg right (Kubernetes is conceptually borg v3 with omega being v2) this is different from how Google runs the things. They'l

by mfer 5y ago

If I've read the Google papers on borg right (Kubernetes is conceptually borg v3 with omega being v2) this is different from how Google runs the things. They'll do warehouse scale computing with borg operating large clusters. borg is at the bottom. The workloads spanning dev, test, and prod then run on these clusters. By having large clusters with lots of things running on them they get high utilization of the hardware and need less hardware. It's amusing to see k8s used in such a different way and one that often uses a lot more hardware while driving up costs. Concepts Google used to lower the cost. Or, maybe I read the papers and book wrong. I like the idea of higher utilization and better efficiency because it uses less resources which is more green.

q3k 5y ago
> Or, maybe I read the papers and book wrong. No, that's exactly how it works. You have clusters spanning a datacenter failure domain (~= an AZ), and everything from prod to dev workloads runs on there, with low priority batch jobs bringing up the resource utilization to a sensible level. You can do the same thing with k8s, you just have to trust its multitenancy support. You have RBAC, priority, quotas, preemption, pod security policies, network policies... Use them. You can even force some workloads to use gVisor or separate prod and dev workloads on different worker machines.
fulafel 5y ago
How do they do version upgrades, isn't that the traditional Achilles heel of K8s that leads people to want to frequently recreate clusters from scratch and/or do blue/green?
- mfer 5y ago
  First, if I understand it right... Google does some smart things in upgrades. They do things like tests and then upgrade their equivalent to AZs in a data center. I'm sure there have been upgrades gone bad that they've had to fix. Kubernetes can be upgraded. I've watched nicely done upgrades happening 4 or 5 years ago. I've watched simple upgrades happen more recently. It's not unheard of. Even in public clouds I've upgrade Kubernetes through many minor versions without issue. I would argue it's more work to create more clusters. You need to migrate workloads and anything pointing to them. It also would cost more as you have to run more hardware.
  - deleted 5y ago
    [deleted]
  - dharmab 5y ago
    We have active clusters that have been continually updated since 1.13 or so. Of course it is the cluster of Theseus since every single bit of compute has been entirely replaced :)
- q3k 5y ago
  If you don't depend on clusters being 100% available all the time and design your applications to handle cluster-wide outages (which you need to do at Google scale anyway), then simply doing progressive rollouts across clusters is good enough. If a cluster rollout fails and knocks some workloads offline, so be it. Just revert the rollout and figure out what went wrong. You can also throw in some smaller clusters that replicate the 'main' clusters' software stack and have some workloads whose downtime does not impact production jobs (not just dev workloads, but replicas of prod workloads!). These can be the earliest in your rollout pipeline and serve as an early stage canary.
- jeffbee 5y ago
  The borg concepts and interfaces have been the same for ages. The borglet and borgmaster get released and pushed out very frequently (daily or weekly) and it doesn't break anything among the workloads. There are not maintenance windows for these changes because containers can run while the borglet restarts, and borgmasters are protocol/format-compatible so the new release binary simply joins the quorum. Machines also get rotated out of the borg cell regularly for kernel upgrades, way more often than I've seen outside of Google. I think an important thing to know about K8s is Omega failed to replace Borg, and then the Omega people created K8s. So K8s does not necessarily descend from Borg, and not all of Borg's desirable attributes made it into K8s.
  - dekhn 5y ago
    The omega folks were on the borgmaster and borglet teams when they were building omega (I was on the borg team at the same time, but working on a different project). It's fair to say that k8s is an intellectual inheritor of the parts of borg that are required for it to be useful on the outside. Borglet/borgmaster releases definitely break workloads. I recall one where something was rolled out to all the test clusters, passed all the tests (except one of ours) and was about to be promoted to increasing percents of the prod clusters. Whilst debugging why our test (which is not part of the feature rollout) broke we realize that if this had rolled out to prod, it would have broken all Tensorflow jobs, and would have been a major OMG. But yeah, most of the time, the release process for borglet and borgmaster is fairly fast and fairly reliable.
    - jeffbee 5y ago
      > required for it to be useful on the outside. A big part of this is that outside Google, the number of people who have to operate k8s as a fraction of all k8s users is way higher than the fraction of borg users who have to operate borg, so there's a lot of stuff in k8s that is 'end user experience' comforts and affordances for operators.
- lokar 5y ago
  Nothing important is single homed to one cluster. Much emphasis is placed on preventing correlated failed of clusters.
- moondev 5y ago
  Highly available kubeadm clusters are designed to be upgraded in place. The Kubernetes api-server is also designed to function with a minor version skew (for example v1.19.x and v1.20.x) that would happen during an in-place upgrade. Cluster API takes the above and can in-place upgrade clusters for you. It's pretty awesome to see first hand. Bumping the Kubernetes version string and machine image reference can upgrade and even swap distros of a running cluster with no down-time.
  - dharmab 5y ago
    One caveat is that downgrading the apiserver is not guaranteed to be possible, since the schema of some types in the API may have been migrated to newer versions in the etcd database that the previous version may not be able to read. There are tools such as Velero (https://velero.io https://velero.io) which can restore a previous snapshot, but you will likely incur downtime and lose any changes since the snapshot.
lifty 5y ago
You’re right, vanilla Kubernetes has a level of complexity that starts paying off at a certain cluster level. But the wide adoption of K8s also shows that people love the standardization and API it offers for orchestrating workloads even if they don’t take advantage of its scaling capabilities. My hope is that projects like k3s will manage to cover that small scale spectrum of the market.
- void_mint 5y ago
  I've intentionally ignored Kubernetes for a few years. Is it worth looking into K3s instead?
  - koeng 5y ago
    I've really enjoyed developing on K3s, would recommend.
  - antonvs 5y ago
    It depends what you're looking for. One attraction of k3s is you can get a "real" production-grade K8s running on a single machine, or a small cluster, very quickly and easily. That can be great for learning, development and testing, certainly. K3s is actually targeted at "edge" scenarios where you aren't running in a cloud, and don't have the ability and/or desire to dynamically adjust the cluster size. You can scale a k3s cluster easily enough manually, adding or deleting nodes, but to get cluster autoscaling you'd need to do environment-dependent work to make it automatic. If you do want that, then things will likely be quite a bit easier with a managed cluster like Google's GKE, or even possibly an self-installed cluster using a tool like kops, kubespray etc. (It's a while since I've used any of those, so not sure which is best.) AWS EKS is another choice, although its setup is a bit more complex and I wouldn't really recommend it to someone getting started. And for production scenarios, integration with cloud load balancers, network environment etc. is similarly going to be easier with a provider-managed cluster. It's all possible to do with k3s, but it's more work and a steeper learning curve.
andrewstuart2 5y ago
And furthermore, the primary problem with scaling out the number of clusters is that it hamstrings one of the primary value propositions of kubernetes: increased utilization. The scheduler can't do its job without yet another scheduler on top if you spread your workloads outside of their sphere of influence.
pyuser583 5y ago
Sounds like a Mesos-style arrangement.
closeparen 5y ago
Exactly. We practice this with Mesos. The point is that a central infrastructure team maintains the (regional) clusters and provides an interface for application teams to submit the services. Each application team maintaining its own cluster or dealing directly with the full power of the cluster scheduler ecosystem would defeat the purpose.
bambambazooka 5y ago
Could you please name the papers and the book?
- q3k 5y ago
  Not GP, but: https://research.google/pubs/pub43438/ https://research.google/pubs/pub43438/ https://sre.google/sre-book/production-environment/ https://sre.google/sre-book/production-environment/ are the best public documentation entrypoints that I know of.
conradev 5y ago
I've also see Kubernetes being used this way, with one cluster per data center for company-wide utilization with segmentation being at the _namespace_ layer. The priority class system is heavily utilized to make sure production workloads are always running, and other workloads are pre-empted as needed.
atmosx 5y ago
> It's amusing to see k8s used in such a different way and one that often uses a lot more hardware while driving up costs. To be fair, google runs its own datacenters with teams doing research & optimisations on all levels of the stack: hardware and software and most importantly an amount of engineering resources converging to infinity. The rest are stuck with VMs that share network interfaces and have to monitor CPU steal, understand complex pricing models, etc. Engineering resources are scarce so most companies will over-provision just to be safe and because profiling the application and fixing that API call that takes too long is expensive, will spin-up another 50 pods.

by mfer 5y ago

q3k 5y ago

> Or, maybe I read the papers and book wrong. No, that's exactly how it works. You have clusters spanning a datacenter failure domain (~= an AZ), and everything from prod to dev workloads runs on there, with low priority batch jobs bringing up the resource utilization to a sensible level. You can do the same thing with k8s, you just have to trust its multitenancy support. You have RBAC, priority, quotas, preemption, pod security policies, network policies... Use them. You can even force some workloads to use gVisor or separate prod and dev workloads on different worker machines.

fulafel 5y ago

How do they do version upgrades, isn't that the traditional Achilles heel of K8s that leads people to want to frequently recreate clusters from scratch and/or do blue/green?

mfer 5y ago
First, if I understand it right... Google does some smart things in upgrades. They do things like tests and then upgrade their equivalent to AZs in a data center. I'm sure there have been upgrades gone bad that they've had to fix. Kubernetes can be upgraded. I've watched nicely done upgrades happening 4 or 5 years ago. I've watched simple upgrades happen more recently. It's not unheard of. Even in public clouds I've upgrade Kubernetes through many minor versions without issue. I would argue it's more work to create more clusters. You need to migrate workloads and anything pointing to them. It also would cost more as you have to run more hardware.
- deleted 5y ago
  [deleted]
- dharmab 5y ago
  We have active clusters that have been continually updated since 1.13 or so. Of course it is the cluster of Theseus since every single bit of compute has been entirely replaced :)
q3k 5y ago
If you don't depend on clusters being 100% available all the time and design your applications to handle cluster-wide outages (which you need to do at Google scale anyway), then simply doing progressive rollouts across clusters is good enough. If a cluster rollout fails and knocks some workloads offline, so be it. Just revert the rollout and figure out what went wrong. You can also throw in some smaller clusters that replicate the 'main' clusters' software stack and have some workloads whose downtime does not impact production jobs (not just dev workloads, but replicas of prod workloads!). These can be the earliest in your rollout pipeline and serve as an early stage canary.
jeffbee 5y ago
The borg concepts and interfaces have been the same for ages. The borglet and borgmaster get released and pushed out very frequently (daily or weekly) and it doesn't break anything among the workloads. There are not maintenance windows for these changes because containers can run while the borglet restarts, and borgmasters are protocol/format-compatible so the new release binary simply joins the quorum. Machines also get rotated out of the borg cell regularly for kernel upgrades, way more often than I've seen outside of Google. I think an important thing to know about K8s is Omega failed to replace Borg, and then the Omega people created K8s. So K8s does not necessarily descend from Borg, and not all of Borg's desirable attributes made it into K8s.
- dekhn 5y ago
  The omega folks were on the borgmaster and borglet teams when they were building omega (I was on the borg team at the same time, but working on a different project). It's fair to say that k8s is an intellectual inheritor of the parts of borg that are required for it to be useful on the outside. Borglet/borgmaster releases definitely break workloads. I recall one where something was rolled out to all the test clusters, passed all the tests (except one of ours) and was about to be promoted to increasing percents of the prod clusters. Whilst debugging why our test (which is not part of the feature rollout) broke we realize that if this had rolled out to prod, it would have broken all Tensorflow jobs, and would have been a major OMG. But yeah, most of the time, the release process for borglet and borgmaster is fairly fast and fairly reliable.
  - jeffbee 5y ago
    > required for it to be useful on the outside. A big part of this is that outside Google, the number of people who have to operate k8s as a fraction of all k8s users is way higher than the fraction of borg users who have to operate borg, so there's a lot of stuff in k8s that is 'end user experience' comforts and affordances for operators.
lokar 5y ago
Nothing important is single homed to one cluster. Much emphasis is placed on preventing correlated failed of clusters.
moondev 5y ago
Highly available kubeadm clusters are designed to be upgraded in place. The Kubernetes api-server is also designed to function with a minor version skew (for example v1.19.x and v1.20.x) that would happen during an in-place upgrade. Cluster API takes the above and can in-place upgrade clusters for you. It's pretty awesome to see first hand. Bumping the Kubernetes version string and machine image reference can upgrade and even swap distros of a running cluster with no down-time.
- dharmab 5y ago
  One caveat is that downgrading the apiserver is not guaranteed to be possible, since the schema of some types in the API may have been migrated to newer versions in the etcd database that the previous version may not be able to read. There are tools such as Velero (https://velero.io https://velero.io) which can restore a previous snapshot, but you will likely incur downtime and lose any changes since the snapshot.

lifty 5y ago

You’re right, vanilla Kubernetes has a level of complexity that starts paying off at a certain cluster level. But the wide adoption of K8s also shows that people love the standardization and API it offers for orchestrating workloads even if they don’t take advantage of its scaling capabilities. My hope is that projects like k3s will manage to cover that small scale spectrum of the market.

void_mint 5y ago
I've intentionally ignored Kubernetes for a few years. Is it worth looking into K3s instead?
- koeng 5y ago
  I've really enjoyed developing on K3s, would recommend.
- antonvs 5y ago
  It depends what you're looking for. One attraction of k3s is you can get a "real" production-grade K8s running on a single machine, or a small cluster, very quickly and easily. That can be great for learning, development and testing, certainly. K3s is actually targeted at "edge" scenarios where you aren't running in a cloud, and don't have the ability and/or desire to dynamically adjust the cluster size. You can scale a k3s cluster easily enough manually, adding or deleting nodes, but to get cluster autoscaling you'd need to do environment-dependent work to make it automatic. If you do want that, then things will likely be quite a bit easier with a managed cluster like Google's GKE, or even possibly an self-installed cluster using a tool like kops, kubespray etc. (It's a while since I've used any of those, so not sure which is best.) AWS EKS is another choice, although its setup is a bit more complex and I wouldn't really recommend it to someone getting started. And for production scenarios, integration with cloud load balancers, network environment etc. is similarly going to be easier with a provider-managed cluster. It's all possible to do with k3s, but it's more work and a steeper learning curve.

andrewstuart2 5y ago

And furthermore, the primary problem with scaling out the number of clusters is that it hamstrings one of the primary value propositions of kubernetes: increased utilization. The scheduler can't do its job without yet another scheduler on top if you spread your workloads outside of their sphere of influence.

pyuser583 5y ago

Sounds like a Mesos-style arrangement.

closeparen 5y ago

Exactly. We practice this with Mesos. The point is that a central infrastructure team maintains the (regional) clusters and provides an interface for application teams to submit the services. Each application team maintaining its own cluster or dealing directly with the full power of the cluster scheduler ecosystem would defeat the purpose.

bambambazooka 5y ago

Could you please name the papers and the book?

q3k 5y ago
Not GP, but: https://research.google/pubs/pub43438/ https://research.google/pubs/pub43438/ https://sre.google/sre-book/production-environment/ https://sre.google/sre-book/production-environment/ are the best public documentation entrypoints that I know of.

conradev 5y ago

I've also see Kubernetes being used this way, with one cluster per data center for company-wide utilization with segmentation being at the _namespace_ layer. The priority class system is heavily utilized to make sure production workloads are always running, and other workloads are pre-empted as needed.

atmosx 5y ago

> It's amusing to see k8s used in such a different way and one that often uses a lot more hardware while driving up costs. To be fair, google runs its own datacenters with teams doing research & optimisations on all levels of the stack: hardware and software and most importantly an amount of engineering resources converging to infinity. The rest are stuck with VMs that share network interfaces and have to monitor CPU steal, understand complex pricing models, etc. Engineering resources are scarce so most companies will over-provision just to be safe and because profiling the application and fixing that API call that takes too long is expensive, will spin-up another 50 pods.