I have 2: 1. We build our own custom build system, because there is no CI that can do actual DAGs (maybe a few). A custom Kubernetes operator that parses Jsonn

I have 2: 1. We build our own custom build system, because there is no CI that can do actual DAGs (maybe a few). A custom Kubernetes operator that parses Jsonnet files to create 100s of CRDs and pods to achieve extreme parallelization. EKS was 144$/mo (now 72$) but no info on master node types. Using watch endpoints with hundreds of pods did not scale well. They had to bump up the master node instances to c5.18xlarge, but same price for managed. But figuring out it was needed to do just scale-up took days. One c5.18xlarge is 2k$ month, and EKS runs at least 3 for HA. So it's a horror story for them. But we also had 100s of worker nodes so it might offset some of them. 2. Similar to CI, we allowed devs to deploy all microservices (~80) from any branch so that they can port-forward and use them. All of them had Ingress endpoints. Days after headaches and frustrations, it turns out nginx ingress generates megabytes of configuration whenever a new deployment occurs, forks a new subprocess with new cfg, kills the other connections. When it's done often, it takes 30GB of memory when 50 developers use it (~4000 pods) and it often dies and restarts. Similar story for Prometheus, kube-state-metrics; they do not like short-lived containers and hug on memory.