6 ms·
I'm not a huge fan of Kubernetes. However, I think there are some great use cases and undeniably some super intelligent people pushing it to amazing limits. Ho
by mrits 4y ago
I'm not a huge fan of Kubernetes. However, I think there are some great use cases and undeniably some super intelligent people pushing it to amazing limits.
However, after reading over this there are some serious red flags. I wonder if this team even understands what alternatives there are for scheduling at this scale or the real trade offs. It seems like an average choice at best and if I was paying the light bill I'd definitely object to going this route.
- Thaxll 4y agoThere is no alternative as far as I know. Which open source or private solution scale above 10k nodes and 100k apps ( pods )?
- emmp 4y agoOne of Nomad's major pitches is that it can scale larger than K8s. It's all over any comparison between the two.
- dharmab 4y agoAnd HashiCorp has a lot of success stories and experience making it work well. They can help you set up a continent-spanning cluster with 100k+ nodes.
- rco8786 4y agoMesos definitely does
- fmajid 4y agoHPC schedulers, most likely.
- fancy_pantser 4y agoIn HPC we see Slurm pretty often.
- dekhn 4y agoCan you use SLURM to schedule services? IE, long-running not-batch-jobs that expose network ports (often multiple)? That's what k8s (and borg) really are- service schedules, not batch schedulers.
- prpl 4y agoYou can schedule jobs without a wall clock time if the admin didn’t disable it, but it’s awkward. That will make scheduling MPI jobs hard/impossible.
- __turbobrew__ 4y agoRun your batch jobs in slurm and your deployments in gke.
- dekhn 4y agoI can run batch jobs in GKE. It looks like SLURM doesn't have a way to handle endpoints (IE, a port manager) so I agree, deployments will have to stay in something like k8s.
- dharmab 4y agoThere are many private, proprietary systems that exceed those scales. They are usually bespoke to the applications that they run. I work with two such systems and know of others in finance, energy and scientific computing. Not to mention Borg at Google. In the commercial proprietary world, clustered mainframes and supercomputers have addressed this niche for decades. In the open source world, HashiCorp Nomad is the most analogous alternative to Kubernetes on commodity hardware, while SLURM is very successful for supercomputing. I've also scale tested k8s to 15k nodes (in a limited configuration for a single application). At that point we ran out of underlying hardware budgeted for the test.
- FridgeSeal 4y ago> There are many private, proprietary systems that exceed those scales Yeah, and then you have to run (and pay for), some private, proprietary system. K8s might be a bit worse for this, but it’s at least “commodity tooling” and lots of common tooling supports it out of the box.
- dharmab 4y agoUsually the private proprietary systems are used because the application is at least one order of magnitude larger in scale than k8s can support.
- smueller1234 4y agoThey also sometimes predate k8s and migration is expensive.
- Thaxll 4y agoHPC does a fraction of what k8s does though? Can you run 100k never ending jobs listening to networking? From what I know HPC is basically run that short lived compute model on a very large cluster that's it.
- eep_social 4y ago> Borg at Google Which is what Kubernetes is based on.
- IceWreck 4y ago> private solution whatever Google and Facebook use internally.