6 ms·

How much different/better/worse is this compared to Kubernetes?

fh973 7y ago
Much simpler to use as a user. I'd say the k8s API is very much over designed and has artifacts that you don't find in other job description languages. Or can anyone explain me the need for StorageClasses, PVCs and PVs with CSI mixins just to access some files? As a scheduler it's much more powerful in capabilities, scalability and performance. It was designed to react to different workload demands in huge clusters very quickly. Imagine a data center goes offline and you need to spin up the search engine while killing a map reduce with 20k workers. A borg cluster is war zone, priorities and quotas win.
- jrockway 7y ago
  At Google, a vast majority of jobs persisted data ("wrote files") over an RPC API, so the container orchestrator didn't have to care about storage. Over in the real world, that seems to be extremely uncommon. Best case: cloud providers give you a block device tied to an AZ, and you're on your own (or you rewrite your app to use S3, or something that implements a similar API). On prem, people use lots of crazy things ranging from NFS to multi-million-dollar black boxes. Kubernetes has to support them all. CSIs exist because there are a billion vendors of "cloud storage" solutions, and apps need a way to decouple themselves from the details. If you just need a block device, your Kubernetes app can run against anything that implements that API; your app you developed on DigitalOcean will Just Work on GKE. That's the idea behind those. For workload management, Borg feels very close to StatefulSets; your replica count could increase and decrease as you demanded it, but each task was individually named (0.your-job.cell). I ran all my jobs with semantics similar to Deployments (tasks did not care which id they had; they all showed up as load balancer targets) and I think Borg had different abstractions for things in Kubernetes like ReplicaSets (i.e., a particular [config,code] tuple that is running in production that logically makes up a deployment). You didn't ask so I won't go into the details as it gets very complicated, but networking is vastly different. (Kubernetes tries to provide IPAM and connection-balancing that is transparent to workloads; Google preferred client-side "smart" load-balancing libraries. That meant that load-balancing complexity was outside the scope of Borg, but for Kubernetes, it's very much in scope. Often with confusing results, but I digress.) My experience having used both extensively is that there are about the same to the end user. Before I worked at Google, I always let "someone else" handle deployment and maintenance of production. At Google, I felt like I could write and release a piece of software to Borg in the same day (and did once!) After Google, I struggled with a bunch of orchestrators until I landed on Kubernetes, which gives me similar confidence and ease of use. Kubernetes has more API surface, probably because it has more users. At Google you could declare by fiat how jobs were to be run. In the real world, people aren't going to use your thing if you don't support their favorite feature. I have tried the opinionated container orchestrators in the real world (Heroku, Convox, ECS) and they didn't make me happy, while Kubernetes did.
- ithkuil 7y ago
  while I do agree that k8s is quite overengineered you're not comparing apples to apples: on Borg you don't have persistent storage, because that's solved at the application level with libraries that talk directly to Colossus etc. A lot of complexity in k8s is the consequences of making it adaptable to different environments
jeffbee 7y ago
k8s and borg are similar in the same way as gRPC and stubby: very similar in conceptual architecture, quite different in objective outcomes. Read the linked article to see that Borg cells have over ten thousand machines each, then note that k8s falls apart with 5000 machines. k8s supports up to 100 pods per node, and you query this trace data to see how that compares to borg. k8s in my experience can schedule about 5 pods per second, and this paper gives much higher figures for borg.
- polskibus 7y ago
  Why is the community getting such a bad deal then? Why does kubernetes have to be so much worse than Borg? Is it time to rewrite Kubernetes?
  - lokar 7y ago
    It's not worse overall, just different. Early on (changing somewhat now), large scale and throughput were not big goals for K8S. Given most uses are with VMs, and pretty much all uses are tiny compared to Borg, this was a totally reasonable prioritization. K8S fixes many many serious issues in Borg that are mostly unfixable at this point (I worked on Borg for many years), it's better in many ways. For scale specifically, Borg has a big advantage in that the state database in internal (the master maintains it directly on a local disk). This is naturally much faster then using etcd (at least etcd today).
  - jeffbee 7y ago
    That seems a bit unfair. The community is getting a pretty square deal with k8s, few people need to manage cells with more than 5000 machines, and those that do can probably afford to write their own schedulers that meet their scalability and performance needs. Google can't just throw Borg over the wall because it's pretty hairy, relies on other parts of Google that also aren't public, and because of the general belief that "the community" is not ready to deal with a large, high-performance C++ program of the type that Google tends to write, which is partly condescension but also pretty much true.
    - 6y ago

Much simpler to use as a user. I'd say the k8s API is very much over designed and has artifacts that you don't find in other job description languages. Or can anyone explain me the need for StorageClasses, PVCs and PVs with CSI mixins just to access some files? As a scheduler it's much more powerful in capabilities, scalability and performance. It was designed to react to different workload demands in huge clusters very quickly. Imagine a data center goes offline and you need to spin up the search engine while killing a map reduce with 20k workers. A borg cluster is war zone, priorities and quotas win.

jrockway 7y ago
At Google, a vast majority of jobs persisted data ("wrote files") over an RPC API, so the container orchestrator didn't have to care about storage. Over in the real world, that seems to be extremely uncommon. Best case: cloud providers give you a block device tied to an AZ, and you're on your own (or you rewrite your app to use S3, or something that implements a similar API). On prem, people use lots of crazy things ranging from NFS to multi-million-dollar black boxes. Kubernetes has to support them all. CSIs exist because there are a billion vendors of "cloud storage" solutions, and apps need a way to decouple themselves from the details. If you just need a block device, your Kubernetes app can run against anything that implements that API; your app you developed on DigitalOcean will Just Work on GKE. That's the idea behind those. For workload management, Borg feels very close to StatefulSets; your replica count could increase and decrease as you demanded it, but each task was individually named (0.your-job.cell). I ran all my jobs with semantics similar to Deployments (tasks did not care which id they had; they all showed up as load balancer targets) and I think Borg had different abstractions for things in Kubernetes like ReplicaSets (i.e., a particular [config,code] tuple that is running in production that logically makes up a deployment). You didn't ask so I won't go into the details as it gets very complicated, but networking is vastly different. (Kubernetes tries to provide IPAM and connection-balancing that is transparent to workloads; Google preferred client-side "smart" load-balancing libraries. That meant that load-balancing complexity was outside the scope of Borg, but for Kubernetes, it's very much in scope. Often with confusing results, but I digress.) My experience having used both extensively is that there are about the same to the end user. Before I worked at Google, I always let "someone else" handle deployment and maintenance of production. At Google, I felt like I could write and release a piece of software to Borg in the same day (and did once!) After Google, I struggled with a bunch of orchestrators until I landed on Kubernetes, which gives me similar confidence and ease of use. Kubernetes has more API surface, probably because it has more users. At Google you could declare by fiat how jobs were to be run. In the real world, people aren't going to use your thing if you don't support their favorite feature. I have tried the opinionated container orchestrators in the real world (Heroku, Convox, ECS) and they didn't make me happy, while Kubernetes did.
ithkuil 7y ago
while I do agree that k8s is quite overengineered you're not comparing apples to apples: on Borg you don't have persistent storage, because that's solved at the application level with libraries that talk directly to Colossus etc. A lot of complexity in k8s is the consequences of making it adaptable to different environments

jeffbee 7y ago

k8s and borg are similar in the same way as gRPC and stubby: very similar in conceptual architecture, quite different in objective outcomes. Read the linked article to see that Borg cells have over ten thousand machines each, then note that k8s falls apart with 5000 machines. k8s supports up to 100 pods per node, and you query this trace data to see how that compares to borg. k8s in my experience can schedule about 5 pods per second, and this paper gives much higher figures for borg.

polskibus 7y ago
Why is the community getting such a bad deal then? Why does kubernetes have to be so much worse than Borg? Is it time to rewrite Kubernetes?
- lokar 7y ago
  It's not worse overall, just different. Early on (changing somewhat now), large scale and throughput were not big goals for K8S. Given most uses are with VMs, and pretty much all uses are tiny compared to Borg, this was a totally reasonable prioritization. K8S fixes many many serious issues in Borg that are mostly unfixable at this point (I worked on Borg for many years), it's better in many ways. For scale specifically, Borg has a big advantage in that the state database in internal (the master maintains it directly on a local disk). This is naturally much faster then using etcd (at least etcd today).
- jeffbee 7y ago
  That seems a bit unfair. The community is getting a pretty square deal with k8s, few people need to manage cells with more than 5000 machines, and those that do can probably afford to write their own schedulers that meet their scalability and performance needs. Google can't just throw Borg over the wall because it's pretty hairy, relies on other parts of Google that also aren't public, and because of the general belief that "the community" is not ready to deal with a large, high-performance C++ program of the type that Google tends to write, which is partly condescension but also pretty much true.
  - 6y ago