4 ms·
I was under the impresion that modern AI still runs mainly baremetal. Isn't pytorch parallel distribute not good enough?
by papichulo2023 2y ago
I was under the impresion that modern AI still runs mainly baremetal. Isn't pytorch parallel distribute not good enough?
- deleted 2y ago[deleted]
- htrp 2y agoThere are a ton of tricks you can do to share hbm/flops amongst GPUs in a cluster.
- chuckadams 2y agoYou can use kubernetes to manage bare metal machines, but it probably doesn't make much difference: once the bits are on the GPU, I doubt a properly configured VM is getting in the way, let alone containerd.
- jfkfif 2y agothe problem is multinode runs that communicate through the network
- josh-sematic 2y agoYes, which is especially important for training. Getting good GPU interconnect can be really important for training large models.
- freeone3000 2y agoMultinode runs don’t communicate through the network in a DGX configuration. NVlink allows for RDMA over direct infiniband. No need for network here.
- tomoyoirl 2y agoInfiniband is a network too… But even if we set that aside you’ll get access to your data over a network connection because these are expensive nodes running batch jobs with finite disk space, not personal workstations.
- freeone3000 2y agoYea ofc. Nvidia has for mellanox infiniband, nvlink external, and even mellanox ethernet pci-to-pci, no need to involve the CPU. nvidia-docker has a few mods to support this too.
- throw0101d 2y ago> I was under the impresion that modern AI still runs mainly baremetal. Generally yes, but you can do (PCI) pass-through of GPUs to VMs and containers and such, so it is not strictly necessary.
- 0x457 2y agoYou can give GPU to VM and containers without any overhead these days. However, Nvidia always placed roadblocks when you aren't using their data center hardware in datacenters.
- ipsum2 2y agoThis is purely a scheduler, it doesn't have any any effect on the workload performance.
- btian 2y agonVidia Multi-Instance GPUs has been around for >4 years.