fully agree with the comment. Currently running an >500 node bare metal cluster with GPU nodes in K8s using Mellanox NICs with 80Gbit/s connectivity. Customer

fully agree with the comment. Currently running an >500 node bare metal cluster with GPU nodes in K8s using Mellanox NICs with 80Gbit/s connectivity. Customer is scheduling work load with Airflow on the K8s farm. Coming from a HPC background, I can confirm your comment. For batch-like workload, K8s and Airflow is a bad choice regarding farm utilization and workload placement. Classic HPC schedulers like Slurm are way more efficient on using farm resources for this kind of workload. Also networking wise is Calico for high speed Infiniband-based networks might be a bad decision. Control plane is too chatty, and slows down spinning up workload. Also do not ever try IP-on-IP encapsulation - it will make your NIC-speed slow, due to CPU-bound encapsulation. With Mellanox-class NICS, strive for SDN technology or use network transport modes with hardware acceleration on the NIC, like VLAN-tagging.