3 ms·

Why is the community getting such a bad deal then? Why does kubernetes have to be so much worse than Borg? Is it time to rewrite Kubernetes?

lokar 7y ago
It's not worse overall, just different. Early on (changing somewhat now), large scale and throughput were not big goals for K8S. Given most uses are with VMs, and pretty much all uses are tiny compared to Borg, this was a totally reasonable prioritization. K8S fixes many many serious issues in Borg that are mostly unfixable at this point (I worked on Borg for many years), it's better in many ways. For scale specifically, Borg has a big advantage in that the state database in internal (the master maintains it directly on a local disk). This is naturally much faster then using etcd (at least etcd today).
jeffbee 7y ago
That seems a bit unfair. The community is getting a pretty square deal with k8s, few people need to manage cells with more than 5000 machines, and those that do can probably afford to write their own schedulers that meet their scalability and performance needs. Google can't just throw Borg over the wall because it's pretty hairy, relies on other parts of Google that also aren't public, and because of the general belief that "the community" is not ready to deal with a large, high-performance C++ program of the type that Google tends to write, which is partly condescension but also pretty much true.
- mkj 6y ago
  Does Go vs C++ account for some of the performance difference? My limited experience with Go suggests it's way more noisy with strace, which can't be great for throughput.
  - thu2111 6y ago
    Go programs are slow compared to C++, the compiler optimises for its own speed and simplicity rather than fast output binaries. But there are big architectural differences too. Borg can split its master across multiple machines for example. The scheduler is a very tightly optimised piece of code. Borg clusters are managed by a central, dedicated specially trained team so there's basically no limit to the operational complexity that a Borg cluster can take on as long as they're all consistent - which they are.

lokar 7y ago

It's not worse overall, just different. Early on (changing somewhat now), large scale and throughput were not big goals for K8S. Given most uses are with VMs, and pretty much all uses are tiny compared to Borg, this was a totally reasonable prioritization. K8S fixes many many serious issues in Borg that are mostly unfixable at this point (I worked on Borg for many years), it's better in many ways. For scale specifically, Borg has a big advantage in that the state database in internal (the master maintains it directly on a local disk). This is naturally much faster then using etcd (at least etcd today).

jeffbee 7y ago

That seems a bit unfair. The community is getting a pretty square deal with k8s, few people need to manage cells with more than 5000 machines, and those that do can probably afford to write their own schedulers that meet their scalability and performance needs. Google can't just throw Borg over the wall because it's pretty hairy, relies on other parts of Google that also aren't public, and because of the general belief that "the community" is not ready to deal with a large, high-performance C++ program of the type that Google tends to write, which is partly condescension but also pretty much true.

mkj 6y ago
Does Go vs C++ account for some of the performance difference? My limited experience with Go suggests it's way more noisy with strace, which can't be great for throughput.
- thu2111 6y ago
  Go programs are slow compared to C++, the compiler optimises for its own speed and simplicity rather than fast output binaries. But there are big architectural differences too. Borg can split its master across multiple machines for example. The scheduler is a very tightly optimised piece of code. Borg clusters are managed by a central, dedicated specially trained team so there's basically no limit to the operational complexity that a Borg cluster can take on as long as they're all consistent - which they are.