4 ms·
I have had the privilege of running a mixed machine learning cluster at almost this scale (5000 nodes). I would never dream of doing it on K8s. Its just not rea
by KaiserPro 6y ago
I have had the privilege of running a mixed machine learning cluster at almost this scale (5000 nodes). I would never dream of doing it on K8s. Its just not really designed to run at the this scale usefully, simply or cheaply. Especially if you have complex jobs(ie you have a bunch of dependencies, run this task before that).
On AWS we used batch. It runs a docker image on a machine. it takes care of all the machine prep, scheduling and error handling.It scales simply, secrets are built in (using parameter store) and you can have EFS/lustre for file access.
You will however need to make a job description language, but that's trivial (mine is a 500 line lambda)
Its not as nice to use as tractor/alfred/deadline, but it works, and doesn't need a team of devops to keep it running optimally.
K8s has some appeal (widespread adoption) but its still not really a HPC workload system. It's networking scheme is just insane (and chatty as fuck, just look at the graphs, almost 10gigs in control traffic, on AWS if you're running in multiple az, thats about $150 an hour just in transfer alone)
- rutthenut 6y agoI was reading about 'Nomad' from Hashicorp, which would seem to do this too
- bend4r 6y agofully agree with the comment. Currently running an >500 node bare metal cluster with GPU nodes in K8s using Mellanox NICs with 80Gbit/s connectivity. Customer is scheduling work load with Airflow on the K8s farm. Coming from a HPC background, I can confirm your comment. For batch-like workload, K8s and Airflow is a bad choice regarding farm utilization and workload placement. Classic HPC schedulers like Slurm are way more efficient on using farm resources for this kind of workload. Also networking wise is Calico for high speed Infiniband-based networks might be a bad decision. Control plane is too chatty, and slows down spinning up workload. Also do not ever try IP-on-IP encapsulation - it will make your NIC-speed slow, due to CPU-bound encapsulation. With Mellanox-class NICS, strive for SDN technology or use network transport modes with hardware acceleration on the NIC, like VLAN-tagging.