5 ms·

>Pods communicate directly with one another on their pod IP addresses with MPI via SSH, not service endpoints. this is super interesting. They are using mpi on

>Pods communicate directly with one another on their pod IP addresses with MPI via SSH, not service endpoints. this is super interesting. They are using mpi on kubernetes for their AI training i suppose. So they are not using anything like kubeflow. Any idea which framework this is ? The state of AI training on kubernetes is not so hot. And this would be a good learning. There is Ray Distributed today that claims a better performance (as well as a better developer experience) than OpenMPI - https://www.usenix.org/system/files/osdi18-moritz.pdf https://www.usenix.org/system/files/osdi18-moritz.pdf wonder why the choices were made as such

benchess 6y ago
Hi, co-author here! We use a pretty standard tech stack of PyTorch + NCCL + MPI. We've used both OpenMPI and MPICH to varying degrees. Kubeflow is interesting, but it solves a slightly different problem of scheduling/coordinating ML workflows on top of Kube. It doesn't get involved with how an ML job communicates within itself cross-node.
- mmq 6y ago
  Probably OP was referring to the MPIOperator, TFOperator, PytorchOperator, ... they are under the Kuberflow org, but can be deployed independently of Kubeflow itself. Several other projects are using those operators to provide similar abstractions you mentioned in your blog post, e.g. Gang scheduling, cross-nodes communication, ... One difference is that these operators use the Kubernetes service interface for communication, generally exposing a headless service for each replica.
  - sandGorgon 6y ago
    @benchess - yes this is what i meant. Using the operator framework. But more generally, MPI over ssh on a large kubernetes deployment is not a very common pattern. Any reason you chose that ? Have you looked at Ray or Torch-Elastic (which seems to be officially supported by AWS, etc as well) https://github.com/pytorch/elastic https://github.com/pytorch/elastic ?
- alculquicondor 6y ago
  Hi, kube-scheduler maintainer here, currently looking into enabling MPI use cases in k8s. We started a discussion in https://github.com/kubeflow/mpi-operator/issues/315 https://github.com/kubeflow/mpi-operator/issues/315
stabbles 6y ago
Mpi over ssh sounds like a bad idea. Typically hpc clusters have a highly optimized vendor libmpi. E.g. cray has an abi compatible libmpi.so to mpich, so you are likely better of replacing the libmpi.so in your container with the vendor version by mounting a bunch of paths from the host. That can be a hook like nvidia has for their gpu support. One container runtime that does this is Sarus: https://github.com/eth-cscs/sarus https://github.com/eth-cscs/sarus

benchess 6y ago

Hi, co-author here! We use a pretty standard tech stack of PyTorch + NCCL + MPI. We've used both OpenMPI and MPICH to varying degrees. Kubeflow is interesting, but it solves a slightly different problem of scheduling/coordinating ML workflows on top of Kube. It doesn't get involved with how an ML job communicates within itself cross-node.

mmq 6y ago
Probably OP was referring to the MPIOperator, TFOperator, PytorchOperator, ... they are under the Kuberflow org, but can be deployed independently of Kubeflow itself. Several other projects are using those operators to provide similar abstractions you mentioned in your blog post, e.g. Gang scheduling, cross-nodes communication, ... One difference is that these operators use the Kubernetes service interface for communication, generally exposing a headless service for each replica.
- sandGorgon 6y ago
  @benchess - yes this is what i meant. Using the operator framework. But more generally, MPI over ssh on a large kubernetes deployment is not a very common pattern. Any reason you chose that ? Have you looked at Ray or Torch-Elastic (which seems to be officially supported by AWS, etc as well) https://github.com/pytorch/elastic https://github.com/pytorch/elastic ?
alculquicondor 6y ago
Hi, kube-scheduler maintainer here, currently looking into enabling MPI use cases in k8s. We started a discussion in https://github.com/kubeflow/mpi-operator/issues/315 https://github.com/kubeflow/mpi-operator/issues/315

stabbles 6y ago

Mpi over ssh sounds like a bad idea. Typically hpc clusters have a highly optimized vendor libmpi. E.g. cray has an abi compatible libmpi.so to mpich, so you are likely better of replacing the libmpi.so in your container with the vendor version by mounting a bunch of paths from the host. That can be a hook like nvidia has for their gpu support. One container runtime that does this is Sarus: https://github.com/eth-cscs/sarus https://github.com/eth-cscs/sarus