3 ms·
>Our biggest jobs run MPI, and all pods within the job are participating in a single MPI communicator. If any of the participating pods die, the entire job halt
by trhway 6y ago
>Our biggest jobs run MPI, and all pods within the job are participating in a single MPI communicator. If any of the participating pods die, the entire job halts and needs to be restarted.
Sounds like a strange approach for ML jobs - i mean you'd expect that all those parallel subtasks aren't each individually midway interconnected with each other, and that the failed subtask can be easily restarted on its own. With thousands of subtasks running in parallel some are bound to fail for whatever reason. Their choice of MPI over HTTP suggests though that they pay a premium for latency and that suggests the subtasks actively cross-communicating, a typical case for MPI.
- jamesblonde 6y agoThis is probably data-parallel training with collective all-reduce (Horovod probably, as they are using MPI). Membership of the ring in Horovod is static - you can't recover from a failed worker. You would need to build a consistent hashing ring (like a DHT), so that workers could identify and agree on failing workers (heartbeats) and evict them. None of those goodies in Horovod yet. The workaround is to have a chief-node do periodic checkpointing of the model weights and epoc/iteration, so that you can recover from the checkpoint if a worker fails.
- cavisne 6y agoThats what makes ML training challenging, the nodes are constantly all-reducing the weights to each other and the step time is in milliseconds. Thats why you see cloud vendors coming out with really beefy individual hosts with lots of GPU's strapped to them, as datacenter networking sucks for these applications.