> That said, strain on the kube-scheduler is spiky. A new job may consist of many hundreds of pods all being created at once, then return to a relatively low ra

> That said, strain on the kube-scheduler is spiky. A new job may consist of many hundreds of pods all being created at once, then return to a relatively low rate of churn. Last I checked, the default scheduler places Pods one at a time. It might be advantageous to use a gang/batch scheduler like kube-batch[0], Poseidon[1] or DCM[2]. Edit: looks like they're already investigating that approach -- > We tried a few things needing a custom scheduler, but ran into edge cases that caused conflicts with how normal pods were scheduled. Kubernetes 1.18 introduced a plugin architecture for the core Kubernetes scheduler, making it much easier to add features like this natively. We recently landed on the Coscheduling plugin as a good way to solve this problem. [0] https://github.com/kubernetes-sigs/kube-batch https://github.com/kubernetes-sigs/kube-batch [1] https://github.com/kubernetes-sigs/poseidon https://github.com/kubernetes-sigs/poseidon [2] https://github.com/vmware/declarative-cluster-management https://github.com/vmware/declarative-cluster-management