3 ms·

>Distributed Training: Gloo is now supported for distributed training jobs. This is very interesting. Can someone talk about the roadmap of pytorch here ? It s

>Distributed Training: Gloo is now supported for distributed training jobs. This is very interesting. Can someone talk about the roadmap of pytorch here ? It seems everyone is kinda rolling their own - Pytorch has a very confusing distribution story - OpenAI runs Pytorch on Kubernetes with handrolled MPI+SSH - https://pytorch.org/tutorials/beginner/dist_overview.html https://pytorch.org/tutorials/beginner/dist_overview.html - https://pytorch.org/docs/stable/distributed.elastic.html https://pytorch.org/docs/stable/distributed.elastic.html - https://pytorch.org/torchx/latest/ https://pytorch.org/torchx/latest/ - https://www.kubeflow.org/docs/components/training/pytorch/ https://www.kubeflow.org/docs/components/training/pytorch/ - Pytorch-Biggraph is specifically using torch.distributed with gloo (with an MPI backend). So here's the question - if ur a 2 person startup that wants to do Pytorch distributed training using one of the cloud-managed EKS/AKS/GKE services... what should you use ?

orbifold 5y ago
The pytorch lightning people have come up with grid.ai, I personally have obtained good results by using pytorch lightning plus slurm on HPC machines. If I were a startup, I would probably try to build my own small HPC cluster, since that is far more cost effective than renting.
- sandGorgon 5y ago
  so most early stage startups get tens of thousands of dollars of free AWS credits. https://aws.amazon.com/activate/ https://aws.amazon.com/activate/ 100K if ur part of a university accelerator. it is far far more efficient (as a proportion of time-to-market) to rent and build on top of services. Kubernetes is where the wider ecosystem is. I dont like it ...but it is what it is. So Grid.ai is something like AWS Sagemaker. I wanted to figure out what someone can use on a readymade kubernetes cluster.
  - another_ 5y ago
    Check out Determined (https://github.com/determined-ai/determined https://github.com/determined-ai/determined). It supports deploying onto k8s and handles running horovod (and soon other dtrain backends), with most of the complexity abstracted behind a few configuration values. Also, it gives you stuff like experiment tracking / hp search (asha) /scheduling / profiling and etc. Disclaimer: I work for Determined.
  - l33tman 5y ago
    I would agree with you if the case was some sort of web service in general, but a GPU server that is only used in-house to run some common pyTorch training jobs is very simple to buy and install and run. Those cloud credits are going to get eaten fast as its a few $/hour, assuming you can get them (those programs are only available if you get your financing from a participating incubator or investor). But yeah if you do get 100k for free from someone you should of course use them first :)
  - orbifold 5y ago
    During my PhD I burned through ~10^5 of compute hours on 4-8 V100 GPU, 128 Core, 1TB+ HPC nodes. Your 10k AWS credits run out really quickly compared to that. And as a startup you are far better of spending some money on a few beefy machines before renting them.
    - sandGorgon 5y ago
      So this thread is a bit of a segue. Even if u rent/buy your machines...my question was "which framework" I definitely don't want to dispute your calculations.. However, I'm still puzzled what to use.
buildbot 5y ago
"- OpenAI runs Pytorch on Kubernetes with handrolled MPI+SSH" Is that published somewhere?

orbifold 5y ago

The pytorch lightning people have come up with grid.ai, I personally have obtained good results by using pytorch lightning plus slurm on HPC machines. If I were a startup, I would probably try to build my own small HPC cluster, since that is far more cost effective than renting.

sandGorgon 5y ago
so most early stage startups get tens of thousands of dollars of free AWS credits. https://aws.amazon.com/activate/ https://aws.amazon.com/activate/ 100K if ur part of a university accelerator. it is far far more efficient (as a proportion of time-to-market) to rent and build on top of services. Kubernetes is where the wider ecosystem is. I dont like it ...but it is what it is. So Grid.ai is something like AWS Sagemaker. I wanted to figure out what someone can use on a readymade kubernetes cluster.
- another_ 5y ago
  Check out Determined (https://github.com/determined-ai/determined https://github.com/determined-ai/determined). It supports deploying onto k8s and handles running horovod (and soon other dtrain backends), with most of the complexity abstracted behind a few configuration values. Also, it gives you stuff like experiment tracking / hp search (asha) /scheduling / profiling and etc. Disclaimer: I work for Determined.
- l33tman 5y ago
  I would agree with you if the case was some sort of web service in general, but a GPU server that is only used in-house to run some common pyTorch training jobs is very simple to buy and install and run. Those cloud credits are going to get eaten fast as its a few $/hour, assuming you can get them (those programs are only available if you get your financing from a participating incubator or investor). But yeah if you do get 100k for free from someone you should of course use them first :)
- orbifold 5y ago
  During my PhD I burned through ~10^5 of compute hours on 4-8 V100 GPU, 128 Core, 1TB+ HPC nodes. Your 10k AWS credits run out really quickly compared to that. And as a startup you are far better of spending some money on a few beefy machines before renting them.
  - sandGorgon 5y ago
    So this thread is a bit of a segue. Even if u rent/buy your machines...my question was "which framework" I definitely don't want to dispute your calculations.. However, I'm still puzzled what to use.

buildbot 5y ago

"- OpenAI runs Pytorch on Kubernetes with handrolled MPI+SSH" Is that published somewhere?