6 ms·

Scaling Kubernetes to 7,500 Nodes

240 points by sillysaurusx 6y ago

sillysaurusx 6y ago
Godda hand it to OpenAI. My opinion about them is slowly reversing. I was nervous when they went full API, but CLIP is a fantastic model that they released for free.
andyxor 6y ago
Excellent engineering but I wish they also worked on AI
sandGorgon 6y ago
>Pods communicate directly with one another on their pod IP addresses with MPI via SSH, not service endpoints. this is super interesting. They are using mpi on kubernetes for their AI training i suppose. So they are not using anything like kubeflow. Any idea which framework this is ? The state of AI training on kubernetes is not so hot. And this would be a good learning. There is Ray Distributed today that claims a better performance (as well as a better developer experience) than OpenMPI - https://www.usenix.org/system/files/osdi18-moritz.pdf https://www.usenix.org/system/files/osdi18-moritz.pdf wonder why the choices were made as such
- benchess 6y ago
  Hi, co-author here! We use a pretty standard tech stack of PyTorch + NCCL + MPI. We've used both OpenMPI and MPICH to varying degrees. Kubeflow is interesting, but it solves a slightly different problem of scheduling/coordinating ML workflows on top of Kube. It doesn't get involved with how an ML job communicates within itself cross-node.
  - mmq 6y ago
    Probably OP was referring to the MPIOperator, TFOperator, PytorchOperator, ... they are under the Kuberflow org, but can be deployed independently of Kubeflow itself. Several other projects are using those operators to provide similar abstractions you mentioned in your blog post, e.g. Gang scheduling, cross-nodes communication, ... One difference is that these operators use the Kubernetes service interface for communication, generally exposing a headless service for each replica.
    - sandGorgon 6y ago
      @benchess - yes this is what i meant. Using the operator framework. But more generally, MPI over ssh on a large kubernetes deployment is not a very common pattern. Any reason you chose that ? Have you looked at Ray or Torch-Elastic (which seems to be officially supported by AWS, etc as well) https://github.com/pytorch/elastic https://github.com/pytorch/elastic ?
  - alculquicondor 6y ago
    Hi, kube-scheduler maintainer here, currently looking into enabling MPI use cases in k8s. We started a discussion in https://github.com/kubeflow/mpi-operator/issues/315 https://github.com/kubeflow/mpi-operator/issues/315
- stabbles 6y ago
  Mpi over ssh sounds like a bad idea. Typically hpc clusters have a highly optimized vendor libmpi. E.g. cray has an abi compatible libmpi.so to mpich, so you are likely better of replacing the libmpi.so in your container with the vendor version by mounting a bunch of paths from the host. That can be a hook like nvidia has for their gpu support. One container runtime that does this is Sarus: https://github.com/eth-cscs/sarus https://github.com/eth-cscs/sarus
chubot 6y ago
A large machine learning job spans many nodes and runs most efficiently when it has access to all of the hardware resources on each node ... So for many of our workloads, a single pod occupies the entire node Hm, why not just use the underlying nodes then, without Kubernetes? Is the underlying cloud that bad at scheduling, and are they keeping the VMs warm all the time? What are they gaining for this indirection? Is it to get a common interface across GCP and other clouds? Bin-packing or fragmentation is not a common problem there’s relatively low strain on the scheduler. That said, strain on the kube-scheduler is spiky
- benchess 6y ago
  Hi! Co-author here. We do keep the nodes running 24/7, so Kubernetes still provides the scheduling to decide which nodes are free or not at any given time. Generally starting a container on a pre-warmed node is still much much faster than booting a VM. Also, some of our servers are bare-metal. EDIT: Also don't discount the rest of the Kubernetes ecosystem. It's more than just a scheduler. It provides configuration, secrets management, healthchecks, self-healing, service discovery, ACLs... there are absolutely other ways to solve each of these things. But when starting from scratch there's a wide field of additional questions to answer, problems to solve.
  - xorcist 6y ago
    Isn't Kubernetes a pretty lousy scheduler when it doesn't take this into consideration? There are a number of schedulers used in high performance computing that should be able to do a better job.
    - AlphaSite 6y ago
      If all you care about is node in use or not in use I think it’s fine. You don’t need anything complex from the scheduler.
    - chubot 6y ago
      Yeah exactly... This seems closer to an HPC problem, not a "cloud" problem. Related comment from 6 months ago about Kubernetes use cases: https://lobste.rs/s/kx1jj4/what_has_your_experience_with_kubernetes#c_qonwag https://lobste.rs/s/kx1jj4/what_has_your_experience_with_kub... Summary: scale has at least 2 different meanings. Scaling in resources doesn't really mean you need Kubernetes. Scaling in terms of workload diversity is a better use case for it. Kubernetes is basically a knockoff of Borg, but Borg is designed (or evolved) to run diverse services (search, maps, gmail, etc.; batch and low latency). Ironically most people who run their own Kube clusters don't seem to have much workload diversity. On the other hand, HPC is usually about scaling in terms of resources: running a few huge jobs on many nodes. A single job will occupy an entire node (and thousands of nodes), which is what's happening here. I've never used these HPC systems but it looks like they are starting to run on the cloud. Kubernetes may still have been a defensible choice for other reasons, but as someone who used Borg for a long time, it's weird what it's turned into. Sort of like protobufs now have a weird "reflection service". Huh? https://aws.amazon.com/blogs/publicsector/tag/htcondor/ https://aws.amazon.com/blogs/publicsector/tag/htcondor/ https://aws.amazon.com/marketplace/pp/Center-for-High-Throughput-Computing-HTCondor-899-/B073WHVRPR https://aws.amazon.com/marketplace/pp/Center-for-High-Throug...
jacques_chester 6y ago
> That said, strain on the kube-scheduler is spiky. A new job may consist of many hundreds of pods all being created at once, then return to a relatively low rate of churn. Last I checked, the default scheduler places Pods one at a time. It might be advantageous to use a gang/batch scheduler like kube-batch[0], Poseidon[1] or DCM[2]. Edit: looks like they're already investigating that approach -- > We tried a few things needing a custom scheduler, but ran into edge cases that caused conflicts with how normal pods were scheduled. Kubernetes 1.18 introduced a plugin architecture for the core Kubernetes scheduler, making it much easier to add features like this natively. We recently landed on the Coscheduling plugin as a good way to solve this problem. [0] https://github.com/kubernetes-sigs/kube-batch https://github.com/kubernetes-sigs/kube-batch [1] https://github.com/kubernetes-sigs/poseidon https://github.com/kubernetes-sigs/poseidon [2] https://github.com/vmware/declarative-cluster-management https://github.com/vmware/declarative-cluster-management
dssdd 6y ago
>Pod network traffic shaping Have you considered EDT-based rate limiting for Pods? This should scale well compared to TBF or HTB. Cilium developers have integrated this natively: https://cilium.io/blog/2020/11/10/cilium-19#bwmanager https://cilium.io/blog/2020/11/10/cilium-19#bwmanager
- benchess 6y ago
  Hi, co-author here. Yes we are excited about the potential of this!
  - asey 6y ago
    Hi Ben, could I ask the size of your team managing these clusters and development efforts?
trhway 6y ago
>Our biggest jobs run MPI, and all pods within the job are participating in a single MPI communicator. If any of the participating pods die, the entire job halts and needs to be restarted. Sounds like a strange approach for ML jobs - i mean you'd expect that all those parallel subtasks aren't each individually midway interconnected with each other, and that the failed subtask can be easily restarted on its own. With thousands of subtasks running in parallel some are bound to fail for whatever reason. Their choice of MPI over HTTP suggests though that they pay a premium for latency and that suggests the subtasks actively cross-communicating, a typical case for MPI.
- jamesblonde 6y ago
  This is probably data-parallel training with collective all-reduce (Horovod probably, as they are using MPI). Membership of the ring in Horovod is static - you can't recover from a failed worker. You would need to build a consistent hashing ring (like a DHT), so that workers could identify and agree on failing workers (heartbeats) and evict them. None of those goodies in Horovod yet. The workaround is to have a chief-node do periodic checkpointing of the model weights and epoc/iteration, so that you can recover from the checkpoint if a worker fails.
- cavisne 6y ago
  Thats what makes ML training challenging, the nodes are constantly all-reducing the weights to each other and the step time is in milliseconds. Thats why you see cloud vendors coming out with really beefy individual hosts with lots of GPU's strapped to them, as datacenter networking sucks for these applications.
boulos 6y ago
Disclosure: I work on Google Cloud (and have worked with Ben and Eric in their role at OpenAI). For folks that find this topic interesting, I think that Maciek’s (GKE PM) talk at NEXT with Twitter on 15,000 node clusters has a ton of great info [1]. Just having lots of nodes is pretty hard, but having lots of nodes and lots of churn is harder. The diagram Maciek puts into the talk and a related blog post is really the key: scale has many dimensions. For OpenAI, they’re making a reasonable choice as operators of a multi tenant cluster (shared amongst researchers). You want to own the other software on the box, do rolling upgrades, and then get out of the way. Many other folks though, really need a much more dynamic system. Watch Maciek’s talk! [1] https://youtu.be/3AAgWBvM5L0 https://youtu.be/3AAgWBvM5L0
matdehaast 6y ago
When we ran HPC CFD models at university we would just use a head node and SLURM to schedule work on machines. It worked really well. I'm not really sure what benefit Kubernetes adds in this case? If anything I feel like it adds more complexity. Would love to hear from the OpenAI folks as to why they chose to use Kubernetes for this problem.
- volta87 6y ago
  From the POV of an HPC cluster user, when using SLURM's `srun` or similar to schedule a job, this now allows you to use `srun --container=<your container>`, and it will start each node where you app run using the container, and make sure MPI, GPUs, etc, all work. If you don't know anything about containers, it probably will be a bit hard to imagine what this buys you, but don't worry, as more clusters start moving towards this model, you'll have to learn about containers at some point. From the POV of the HPC cluster, it means that the `module` system can be replaced with containers, and that can significantly lower the maintenance overhead of the cluster. In a sense, it turns HPC cluster users into HPC cluster maintainers (that have to build their own images, preparing their own environment, etc).
- bend4r 6y ago
  Modern HPC schedulers like Slurm are able to run containerized workloads making the problem of portability less painful. For batch workload K8s is not sufficient. You need a workflow engine on top like Kubeflow or Airflow, so run you jobs. Thus. consider K8s more as a cluster operating software, but not as a fully-enabled workload management like slurm. Furthermore, K8s has so many features you do not need in HPC environments, where high throughput and load occupation is paramount.
KaiserPro 6y ago
I have had the privilege of running a mixed machine learning cluster at almost this scale (5000 nodes). I would never dream of doing it on K8s. Its just not really designed to run at the this scale usefully, simply or cheaply. Especially if you have complex jobs(ie you have a bunch of dependencies, run this task before that). On AWS we used batch. It runs a docker image on a machine. it takes care of all the machine prep, scheduling and error handling.It scales simply, secrets are built in (using parameter store) and you can have EFS/lustre for file access. You will however need to make a job description language, but that's trivial (mine is a 500 line lambda) Its not as nice to use as tractor/alfred/deadline, but it works, and doesn't need a team of devops to keep it running optimally. K8s has some appeal (widespread adoption) but its still not really a HPC workload system. It's networking scheme is just insane (and chatty as fuck, just look at the graphs, almost 10gigs in control traffic, on AWS if you're running in multiple az, thats about $150 an hour just in transfer alone)
- rutthenut 6y ago
  I was reading about 'Nomad' from Hashicorp, which would seem to do this too
- bend4r 6y ago
  fully agree with the comment. Currently running an >500 node bare metal cluster with GPU nodes in K8s using Mellanox NICs with 80Gbit/s connectivity. Customer is scheduling work load with Airflow on the K8s farm. Coming from a HPC background, I can confirm your comment. For batch-like workload, K8s and Airflow is a bad choice regarding farm utilization and workload placement. Classic HPC schedulers like Slurm are way more efficient on using farm resources for this kind of workload. Also networking wise is Calico for high speed Infiniband-based networks might be a bad decision. Control plane is too chatty, and slows down spinning up workload. Also do not ever try IP-on-IP encapsulation - it will make your NIC-speed slow, due to CPU-bound encapsulation. With Mellanox-class NICS, strive for SDN technology or use network transport modes with hardware acceleration on the NIC, like VLAN-tagging.
harpratap 6y ago
The prometheus scaling problem is a real one. It stems from the "scrape everything" mantra that everyone in prometheus community follows, we should be rather opting-in on which metrics are useful and drop everything by default. We need to go the distributed-tracing way, have smart agents near your metric source when gets everything locally and then decides what metrics are useful and finally ship them off for storage. We are kinda halfway there with OpenTelemetry but the adoption hasn't started yet. Very curious to see how you guys managed it.
- dima_vm 6y ago
  Hey, we (Victoria Metrics) solved the scaling problem for you. We've yet to see a customer with data sizes our cluster version couldn't handle. I'd say "Scrape everything" mantra was born out of necessity -- engineer never knows what will be needed in the future. And "everything" didn't come for free either -- someone needed to expose that metric to scrape in the first place. If you drop most metrics by default -- it's much harder to add it back later and waiting for it to collect enough data. Users who take monitoring seriously, figured out long ago that it's much better to hoard everything that comes to mind from the beginning. [1] https://github.com/VictoriaMetrics/ https://github.com/VictoriaMetrics/

240 points by sillysaurusx 6y ago

Godda hand it to OpenAI. My opinion about them is slowly reversing. I was nervous when they went full API, but CLIP is a fantastic model that they released for free.

andyxor 6y ago

Excellent engineering but I wish they also worked on AI

sandGorgon 6y ago

>Pods communicate directly with one another on their pod IP addresses with MPI via SSH, not service endpoints. this is super interesting. They are using mpi on kubernetes for their AI training i suppose. So they are not using anything like kubeflow. Any idea which framework this is ? The state of AI training on kubernetes is not so hot. And this would be a good learning. There is Ray Distributed today that claims a better performance (as well as a better developer experience) than OpenMPI - https://www.usenix.org/system/files/osdi18-moritz.pdf https://www.usenix.org/system/files/osdi18-moritz.pdf wonder why the choices were made as such

benchess 6y ago
Hi, co-author here! We use a pretty standard tech stack of PyTorch + NCCL + MPI. We've used both OpenMPI and MPICH to varying degrees. Kubeflow is interesting, but it solves a slightly different problem of scheduling/coordinating ML workflows on top of Kube. It doesn't get involved with how an ML job communicates within itself cross-node.
- mmq 6y ago
  Probably OP was referring to the MPIOperator, TFOperator, PytorchOperator, ... they are under the Kuberflow org, but can be deployed independently of Kubeflow itself. Several other projects are using those operators to provide similar abstractions you mentioned in your blog post, e.g. Gang scheduling, cross-nodes communication, ... One difference is that these operators use the Kubernetes service interface for communication, generally exposing a headless service for each replica.
  - sandGorgon 6y ago
    @benchess - yes this is what i meant. Using the operator framework. But more generally, MPI over ssh on a large kubernetes deployment is not a very common pattern. Any reason you chose that ? Have you looked at Ray or Torch-Elastic (which seems to be officially supported by AWS, etc as well) https://github.com/pytorch/elastic https://github.com/pytorch/elastic ?
- alculquicondor 6y ago
  Hi, kube-scheduler maintainer here, currently looking into enabling MPI use cases in k8s. We started a discussion in https://github.com/kubeflow/mpi-operator/issues/315 https://github.com/kubeflow/mpi-operator/issues/315
stabbles 6y ago
Mpi over ssh sounds like a bad idea. Typically hpc clusters have a highly optimized vendor libmpi. E.g. cray has an abi compatible libmpi.so to mpich, so you are likely better of replacing the libmpi.so in your container with the vendor version by mounting a bunch of paths from the host. That can be a hook like nvidia has for their gpu support. One container runtime that does this is Sarus: https://github.com/eth-cscs/sarus https://github.com/eth-cscs/sarus

chubot 6y ago

A large machine learning job spans many nodes and runs most efficiently when it has access to all of the hardware resources on each node ... So for many of our workloads, a single pod occupies the entire node Hm, why not just use the underlying nodes then, without Kubernetes? Is the underlying cloud that bad at scheduling, and are they keeping the VMs warm all the time? What are they gaining for this indirection? Is it to get a common interface across GCP and other clouds? Bin-packing or fragmentation is not a common problem there’s relatively low strain on the scheduler. That said, strain on the kube-scheduler is spiky

benchess 6y ago
Hi! Co-author here. We do keep the nodes running 24/7, so Kubernetes still provides the scheduling to decide which nodes are free or not at any given time. Generally starting a container on a pre-warmed node is still much much faster than booting a VM. Also, some of our servers are bare-metal. EDIT: Also don't discount the rest of the Kubernetes ecosystem. It's more than just a scheduler. It provides configuration, secrets management, healthchecks, self-healing, service discovery, ACLs... there are absolutely other ways to solve each of these things. But when starting from scratch there's a wide field of additional questions to answer, problems to solve.
- xorcist 6y ago
  Isn't Kubernetes a pretty lousy scheduler when it doesn't take this into consideration? There are a number of schedulers used in high performance computing that should be able to do a better job.
  - AlphaSite 6y ago
    If all you care about is node in use or not in use I think it’s fine. You don’t need anything complex from the scheduler.
  - chubot 6y ago
    Yeah exactly... This seems closer to an HPC problem, not a "cloud" problem. Related comment from 6 months ago about Kubernetes use cases: https://lobste.rs/s/kx1jj4/what_has_your_experience_with_kubernetes#c_qonwag https://lobste.rs/s/kx1jj4/what_has_your_experience_with_kub... Summary: scale has at least 2 different meanings. Scaling in resources doesn't really mean you need Kubernetes. Scaling in terms of workload diversity is a better use case for it. Kubernetes is basically a knockoff of Borg, but Borg is designed (or evolved) to run diverse services (search, maps, gmail, etc.; batch and low latency). Ironically most people who run their own Kube clusters don't seem to have much workload diversity. On the other hand, HPC is usually about scaling in terms of resources: running a few huge jobs on many nodes. A single job will occupy an entire node (and thousands of nodes), which is what's happening here. I've never used these HPC systems but it looks like they are starting to run on the cloud. Kubernetes may still have been a defensible choice for other reasons, but as someone who used Borg for a long time, it's weird what it's turned into. Sort of like protobufs now have a weird "reflection service". Huh? https://aws.amazon.com/blogs/publicsector/tag/htcondor/ https://aws.amazon.com/blogs/publicsector/tag/htcondor/ https://aws.amazon.com/marketplace/pp/Center-for-High-Throughput-Computing-HTCondor-899-/B073WHVRPR https://aws.amazon.com/marketplace/pp/Center-for-High-Throug...

jacques_chester 6y ago

> That said, strain on the kube-scheduler is spiky. A new job may consist of many hundreds of pods all being created at once, then return to a relatively low rate of churn. Last I checked, the default scheduler places Pods one at a time. It might be advantageous to use a gang/batch scheduler like kube-batch[0], Poseidon[1] or DCM[2]. Edit: looks like they're already investigating that approach -- > We tried a few things needing a custom scheduler, but ran into edge cases that caused conflicts with how normal pods were scheduled. Kubernetes 1.18 introduced a plugin architecture for the core Kubernetes scheduler, making it much easier to add features like this natively. We recently landed on the Coscheduling plugin as a good way to solve this problem. [0] https://github.com/kubernetes-sigs/kube-batch https://github.com/kubernetes-sigs/kube-batch [1] https://github.com/kubernetes-sigs/poseidon https://github.com/kubernetes-sigs/poseidon [2] https://github.com/vmware/declarative-cluster-management https://github.com/vmware/declarative-cluster-management

dssdd 6y ago

>Pod network traffic shaping Have you considered EDT-based rate limiting for Pods? This should scale well compared to TBF or HTB. Cilium developers have integrated this natively: https://cilium.io/blog/2020/11/10/cilium-19#bwmanager https://cilium.io/blog/2020/11/10/cilium-19#bwmanager

benchess 6y ago
Hi, co-author here. Yes we are excited about the potential of this!
- asey 6y ago
  Hi Ben, could I ask the size of your team managing these clusters and development efforts?

trhway 6y ago

>Our biggest jobs run MPI, and all pods within the job are participating in a single MPI communicator. If any of the participating pods die, the entire job halts and needs to be restarted. Sounds like a strange approach for ML jobs - i mean you'd expect that all those parallel subtasks aren't each individually midway interconnected with each other, and that the failed subtask can be easily restarted on its own. With thousands of subtasks running in parallel some are bound to fail for whatever reason. Their choice of MPI over HTTP suggests though that they pay a premium for latency and that suggests the subtasks actively cross-communicating, a typical case for MPI.

jamesblonde 6y ago
This is probably data-parallel training with collective all-reduce (Horovod probably, as they are using MPI). Membership of the ring in Horovod is static - you can't recover from a failed worker. You would need to build a consistent hashing ring (like a DHT), so that workers could identify and agree on failing workers (heartbeats) and evict them. None of those goodies in Horovod yet. The workaround is to have a chief-node do periodic checkpointing of the model weights and epoc/iteration, so that you can recover from the checkpoint if a worker fails.
cavisne 6y ago
Thats what makes ML training challenging, the nodes are constantly all-reducing the weights to each other and the step time is in milliseconds. Thats why you see cloud vendors coming out with really beefy individual hosts with lots of GPU's strapped to them, as datacenter networking sucks for these applications.

boulos 6y ago

Disclosure: I work on Google Cloud (and have worked with Ben and Eric in their role at OpenAI). For folks that find this topic interesting, I think that Maciek’s (GKE PM) talk at NEXT with Twitter on 15,000 node clusters has a ton of great info [1]. Just having lots of nodes is pretty hard, but having lots of nodes and lots of churn is harder. The diagram Maciek puts into the talk and a related blog post is really the key: scale has many dimensions. For OpenAI, they’re making a reasonable choice as operators of a multi tenant cluster (shared amongst researchers). You want to own the other software on the box, do rolling upgrades, and then get out of the way. Many other folks though, really need a much more dynamic system. Watch Maciek’s talk! [1] https://youtu.be/3AAgWBvM5L0 https://youtu.be/3AAgWBvM5L0

matdehaast 6y ago

When we ran HPC CFD models at university we would just use a head node and SLURM to schedule work on machines. It worked really well. I'm not really sure what benefit Kubernetes adds in this case? If anything I feel like it adds more complexity. Would love to hear from the OpenAI folks as to why they chose to use Kubernetes for this problem.

volta87 6y ago
From the POV of an HPC cluster user, when using SLURM's `srun` or similar to schedule a job, this now allows you to use `srun --container=<your container>`, and it will start each node where you app run using the container, and make sure MPI, GPUs, etc, all work. If you don't know anything about containers, it probably will be a bit hard to imagine what this buys you, but don't worry, as more clusters start moving towards this model, you'll have to learn about containers at some point. From the POV of the HPC cluster, it means that the `module` system can be replaced with containers, and that can significantly lower the maintenance overhead of the cluster. In a sense, it turns HPC cluster users into HPC cluster maintainers (that have to build their own images, preparing their own environment, etc).
bend4r 6y ago
Modern HPC schedulers like Slurm are able to run containerized workloads making the problem of portability less painful. For batch workload K8s is not sufficient. You need a workflow engine on top like Kubeflow or Airflow, so run you jobs. Thus. consider K8s more as a cluster operating software, but not as a fully-enabled workload management like slurm. Furthermore, K8s has so many features you do not need in HPC environments, where high throughput and load occupation is paramount.

KaiserPro 6y ago

I have had the privilege of running a mixed machine learning cluster at almost this scale (5000 nodes). I would never dream of doing it on K8s. Its just not really designed to run at the this scale usefully, simply or cheaply. Especially if you have complex jobs(ie you have a bunch of dependencies, run this task before that). On AWS we used batch. It runs a docker image on a machine. it takes care of all the machine prep, scheduling and error handling.It scales simply, secrets are built in (using parameter store) and you can have EFS/lustre for file access. You will however need to make a job description language, but that's trivial (mine is a 500 line lambda) Its not as nice to use as tractor/alfred/deadline, but it works, and doesn't need a team of devops to keep it running optimally. K8s has some appeal (widespread adoption) but its still not really a HPC workload system. It's networking scheme is just insane (and chatty as fuck, just look at the graphs, almost 10gigs in control traffic, on AWS if you're running in multiple az, thats about $150 an hour just in transfer alone)

rutthenut 6y ago
I was reading about 'Nomad' from Hashicorp, which would seem to do this too
bend4r 6y ago
fully agree with the comment. Currently running an >500 node bare metal cluster with GPU nodes in K8s using Mellanox NICs with 80Gbit/s connectivity. Customer is scheduling work load with Airflow on the K8s farm. Coming from a HPC background, I can confirm your comment. For batch-like workload, K8s and Airflow is a bad choice regarding farm utilization and workload placement. Classic HPC schedulers like Slurm are way more efficient on using farm resources for this kind of workload. Also networking wise is Calico for high speed Infiniband-based networks might be a bad decision. Control plane is too chatty, and slows down spinning up workload. Also do not ever try IP-on-IP encapsulation - it will make your NIC-speed slow, due to CPU-bound encapsulation. With Mellanox-class NICS, strive for SDN technology or use network transport modes with hardware acceleration on the NIC, like VLAN-tagging.

harpratap 6y ago

The prometheus scaling problem is a real one. It stems from the "scrape everything" mantra that everyone in prometheus community follows, we should be rather opting-in on which metrics are useful and drop everything by default. We need to go the distributed-tracing way, have smart agents near your metric source when gets everything locally and then decides what metrics are useful and finally ship them off for storage. We are kinda halfway there with OpenTelemetry but the adoption hasn't started yet. Very curious to see how you guys managed it.

dima_vm 6y ago
Hey, we (Victoria Metrics) solved the scaling problem for you. We've yet to see a customer with data sizes our cluster version couldn't handle. I'd say "Scrape everything" mantra was born out of necessity -- engineer never knows what will be needed in the future. And "everything" didn't come for free either -- someone needed to expose that metric to scrape in the first place. If you drop most metrics by default -- it's much harder to add it back later and waiting for it to collect enough data. Users who take monitoring seriously, figured out long ago that it's much better to hoard everything that comes to mind from the beginning. [1] https://github.com/VictoriaMetrics/ https://github.com/VictoriaMetrics/