5 ms·
This analogy is very misleading. Kubernetes is probably never going to run any real workload internally at Google, but gRPC powers all external APIs of Google C
by mehrdada 7y ago
This analogy is very misleading. Kubernetes is probably never going to run any real workload internally at Google, but gRPC powers all external APIs of Google Cloud, and increasing other Google properties (e.g. ads, assistant), used by mobile apps like Duo and Allo, and have some big internal service use cases. The reason Stubby still dominates internally is simply taking lots of time to migrate to gRPC that might be hard to justify, but I do see gRPC being used very widely internally at Google; it’s simply a matter of time. I don’t see that happening to Kubernetes; it’s a joke when compared to Borg.
Google aside, many other companies like Dropbox rely on gRPC extensively to successfully run infrastructure: https://static.sched.com/hosted_files/grpconf19/f7/Courier%20—%20gRPC%20Conf%202019.pdf https://static.sched.com/hosted_files/grpconf19/f7/Courier%2...
- enitihas 7y agoCuriously, what do you find lacking in Kubernetes compared to Borg?
- mehrdada 7y agoScale, for one thing. Kubernetes is not a bad system but it’s not designed to run Google.
- int0x80 7y agoCan you expand on that, please?
- q3k 7y agohttps://kubernetes.io/docs/setup/cluster-large/ https://kubernetes.io/docs/setup/cluster-large/ More specifically, "No more than 5000 nodes" and "No more than 150000 total pods" is fairly limiting to large (Google-large) clusters.
- int0x80 7y agothanks for the answer.
- dilyevsky 7y agoI used to work on one of Borg teams at Google and now run Kubernetes platform. Nearly every Kubernetes component (node, master, networking) will melt down at fraction of Borg scale. It's not even close.
- int0x80 7y agothanks for the answer.
- Thaxll 7y agoWhat google system runs more than 5k tasks on a single cluster?
- dilyevsky 7y agoHeh, you have no idea. Lots and lots of systems. And 5k wouldn't even register there...
- Thaxll 7y agoCan't edit my comment anymore but I meant 5k nodes / 150k pods / tasks.
- the-rc 7y agohttps://github.com/google/cluster-data/blob/master/README.md#cluster-workload-traces https://github.com/google/cluster-data/blob/master/README.md... "ClusterData2011_2 provides data from an 12.5k-machine cell over about a month-long period in May 2011."
- shereadsthenews 7y ago10k replicas in a single cell is the default charge-free quota of every individual engineer at that company. It’s basically zero.
- shereadsthenews 7y agoI guess you’d find that on an rpc-weighted scale Stubby handles several orders of magnitude more traffic than those gRPC endpoints you mentioned.
- deleted 7y ago[deleted]
- mehrdada 7y agoThis may be true today (although even on this metric I’d estimate Kubernetes to be off by some orders of magnitude, bordering zero). My point is there’s a path and plan forward for gRPC adoption and it’s a matter of transition to a new system (which can, admittedly, be very long). For Kubernetes, I don’t think there is a credible path for replacing Borg.
- CydeWeys 7y agoI work at Google and my team has real workloads running on Kubernetes. There's plenty of internal teams that use GCP. Increasingly this might be the direction things are heading.
- mehrdada 7y agoGCP itself is a job on Borg. ;)
- justicezyx 7y agoThat's not true. GCE uses Borg very differently than normal Google internal systems, which you can imagine is quite natural as they are reserving different customers. GCS and other system, in turn, also differ wildly than GCE. when you talk about GCP as a whole, it becomes impossible to summarize in a few statement, and I doubt there is anyone on earth who is capable to describe it coherently even without time constraint.
- mehrdada 7y agoWhat I said (GCP runs on Borg) is absolutely and technically correct, affirmed by your own comment, which highlights the power and flexibility of Borg. The point being no one[1] at Google relies on Kubernetes for raw cluster management capabilities at scale. They might use it for other things that can make deployment more friendly in some scenarios. (This doesn’t make Kubernetes a bad system by any means, just quite different and not a substitute for Borg whereas gRPC is a direct substitute for Stubby). This debate is better argued in your own eng-misc@ and not on a public forum. [1]: no-one that we care. At Google this is obviously always incorrect. There’s always that someone who uses weird things like mongoDB and AWS.
- shereadsthenews 7y agoAnd there's no reason why a small project should not. But nobody is going to move, say, indexing to GCP. And when it comes to power laws the big things are big and the small things are not.