6 ms·

Scale, for one thing. Kubernetes is not a bad system but it’s not designed to run Google.

• int0x80 7y ago
  Can you expand on that, please?
  • q3k 7y ago
    https://kubernetes.io/docs/setup/cluster-large/ https://kubernetes.io/docs/setup/cluster-large/ More specifically, "No more than 5000 nodes" and "No more than 150000 total pods" is fairly limiting to large (Google-large) clusters.
    • int0x80 7y ago
      thanks for the answer.
  • dilyevsky 7y ago
    I used to work on one of Borg teams at Google and now run Kubernetes platform. Nearly every Kubernetes component (node, master, networking) will melt down at fraction of Borg scale. It's not even close.
    • int0x80 7y ago
      thanks for the answer.
• Thaxll 7y ago
  What google system runs more than 5k tasks on a single cluster?
  • dilyevsky 7y ago
    Heh, you have no idea. Lots and lots of systems. And 5k wouldn't even register there...
    • Thaxll 7y ago
      Can't edit my comment anymore but I meant 5k nodes / 150k pods / tasks.
  • the-rc 7y ago
    https://github.com/google/cluster-data/blob/master/README.md#cluster-workload-traces https://github.com/google/cluster-data/blob/master/README.md... "ClusterData2011_2 provides data from an 12.5k-machine cell over about a month-long period in May 2011."
  • shereadsthenews 7y ago
    10k replicas in a single cell is the default charge-free quota of every individual engineer at that company. It’s basically zero.