3 ms·
There are many private, proprietary systems that exceed those scales. They are usually bespoke to the applications that they run. I work with two such systems a
by dharmab 4y ago
There are many private, proprietary systems that exceed those scales. They are usually bespoke to the applications that they run. I work with two such systems and know of others in finance, energy and scientific computing. Not to mention Borg at Google.
In the commercial proprietary world, clustered mainframes and supercomputers have addressed this niche for decades.
In the open source world, HashiCorp Nomad is the most analogous alternative to Kubernetes on commodity hardware, while SLURM is very successful for supercomputing.
I've also scale tested k8s to 15k nodes (in a limited configuration for a single application). At that point we ran out of underlying hardware budgeted for the test.
- FridgeSeal 4y ago> There are many private, proprietary systems that exceed those scales Yeah, and then you have to run (and pay for), some private, proprietary system. K8s might be a bit worse for this, but it’s at least “commodity tooling” and lots of common tooling supports it out of the box.
- dharmab 4y agoUsually the private proprietary systems are used because the application is at least one order of magnitude larger in scale than k8s can support.
- smueller1234 4y agoThey also sometimes predate k8s and migration is expensive.
- Thaxll 4y agoHPC does a fraction of what k8s does though? Can you run 100k never ending jobs listening to networking? From what I know HPC is basically run that short lived compute model on a very large cluster that's it.
- eep_social 4y ago> Borg at Google Which is what Kubernetes is based on.
- jmillikin 4y agoKubernetes is based on Omega, which was an attempt to replace Borg using a more distributed architecture. Borg uses a single master with elections, with the replicas serving as hot spares. Source: I was on Borg-SRE at Google for ~six years, then worked with Kubernetes at Stripe for ~five.
- eep_social 4y agoYup, and while ISTR that Omega did (eventually) get rolled out, no one ever called it anything but Borg so you’re the best kind of correct :)