3 ms·
Running 50k Jobs on Kubernetes
- geerlingguy 6y agoThere's a GitHub repo that goes along with this video here: https://github.com/geerlingguy/50k-k8s-jobs https://github.com/geerlingguy/50k-k8s-jobs One thing I discovered (while debugging the problems I was having with deploying 50,000 Jobs in one go, without manual garbage collection) was that it seems the default Linode Kubernetes Engine control plane may have 2 GB of RAM, and is likely not set up in an HA cluster (which is how they can make the control plane free). I was debugging the issue through this kubernetes/kubernetes GitHub issue: https://github.com/kubernetes/kubernetes/issues/95492 https://github.com/kubernetes/kubernetes/issues/95492 Even if you have enough RAM, running Kubernetes Jobs (in large numbers) seems to result in a slow but steady increase in time-to-complete: https://github.com/kubernetes/kubernetes/issues/95492#issuecomment-707463968 https://github.com/kubernetes/kubernetes/issues/95492#issuec...