3 ms·

I don't think NCCL + Deepspeed/FSDP are really an alternative to Scuda, as they all require the models in question to be designed for distributed training. They

I don't think NCCL + Deepspeed/FSDP are really an alternative to Scuda, as they all require the models in question to be designed for distributed training. They also require a lot of support in the libraries being used. This has been a struggle for data scientists for a while now. I haven't seen a good solution to allow a data scientist to work locally, but utilize GPUs remotely, without basically just developing remotely (through a VM or Jupyter), or submitting remote jobs (through SLURM or a library specific Kubernetes integration). Scuda is an interesting step towards a better solution for utilizing remote GPUs easily across a wide range of libraries, not just Pytorch and Tensorflow.

seattleeng 2y ago
Why is working locally important?
- theossuary 2y ago
  Working locally still matters, and this is from someone who normally works in tmux/nvim. When working on vision and 3D ML work, being able to quickly open a visualizer windows is imperative to understanding what's going on. For Gaussian Splatting, point cloud work, SLAM, etc. you have to have access to a desktop environment to see visualizations; they very rarely work well remotely (even if they have some Jupyter support). Working remotely, when having to use a desktop environment is painful, no matter the technology. The best tice come up with is using tmux/vim and sunshine/moonlight, but even still I'd rather just have access to everything locally.
kkielhofner 2y ago
> I don't think NCCL + Deepspeed/FSDP are really an alternative to Scuda I put "remote" in quotes because they're not direct equivalents but from a practical standpoint it's the alternate current approach. > they all require the models in question to be designed for distributed training. They also require a lot of support in the libraries being used. IME this has changed quite a bit. Between improved support for torch FSDP, Deepspeed, and especially HF Accelerate wrapping of each with transformer models it's been a while since I've had to put much (if any) work in. That said if you're running random training scripts it likely won't "just work" but given larger models becoming more common I see a lot more torchrun, accelerate, deepspeed, etc in READMEs and code. > This has been a struggle for data scientists for a while now. I haven't seen a good solution to allow a data scientist to work locally, but utilize GPUs remotely, without basically just developing remotely (through a VM or Jupyter) Remotely, as in over the internet? 400gb ethernet is already too slow vs PCIe5 x16 (forget SXM). A 10gb internet connection is 40x slower (plus latency impacts). Remote development via the internet with scuda would be impossibly and completely uselessly slow.

seattleeng 2y ago

Why is working locally important?

theossuary 2y ago
Working locally still matters, and this is from someone who normally works in tmux/nvim. When working on vision and 3D ML work, being able to quickly open a visualizer windows is imperative to understanding what's going on. For Gaussian Splatting, point cloud work, SLAM, etc. you have to have access to a desktop environment to see visualizations; they very rarely work well remotely (even if they have some Jupyter support). Working remotely, when having to use a desktop environment is painful, no matter the technology. The best tice come up with is using tmux/vim and sunshine/moonlight, but even still I'd rather just have access to everything locally.

kkielhofner 2y ago

> I don't think NCCL + Deepspeed/FSDP are really an alternative to Scuda I put "remote" in quotes because they're not direct equivalents but from a practical standpoint it's the alternate current approach. > they all require the models in question to be designed for distributed training. They also require a lot of support in the libraries being used. IME this has changed quite a bit. Between improved support for torch FSDP, Deepspeed, and especially HF Accelerate wrapping of each with transformer models it's been a while since I've had to put much (if any) work in. That said if you're running random training scripts it likely won't "just work" but given larger models becoming more common I see a lot more torchrun, accelerate, deepspeed, etc in READMEs and code. > This has been a struggle for data scientists for a while now. I haven't seen a good solution to allow a data scientist to work locally, but utilize GPUs remotely, without basically just developing remotely (through a VM or Jupyter) Remotely, as in over the internet? 400gb ethernet is already too slow vs PCIe5 x16 (forget SXM). A 10gb internet connection is 40x slower (plus latency impacts). Remote development via the internet with scuda would be impossibly and completely uselessly slow.