5 ms·

This is very interesting but many of the motivations listed are far better served with alternate approaches. For "remote" model training there is NCCL + Deepsp

This is very interesting but many of the motivations listed are far better served with alternate approaches. For "remote" model training there is NCCL + Deepspeed/FSDP/etc. For remote inferencing there are solutions like Triton Inference Server[0] that can do very high-performance hosting of any model for inference. For LLMs specifically there are nearly countless implementations. That said, the ability to use this for testing is interesting but I wonder about GPU contention and as others have noted the performance of such a solution will be terrible even with relatively high speed interconnect (100/400gb ethernet, etc). NCCL has been optimized to support DMA directly between network interfaces and GPUs which is of course considerably faster than solutions like this. Triton can also make use of shared memory, mmap, NCCL, MPI, etc which is one of the many tricks it uses for very performant inference - even across multiple chassis over another network layer. [0] - https://github.com/triton-inference-server/server https://github.com/triton-inference-server/server

theossuary 2y ago
I don't think NCCL + Deepspeed/FSDP are really an alternative to Scuda, as they all require the models in question to be designed for distributed training. They also require a lot of support in the libraries being used. This has been a struggle for data scientists for a while now. I haven't seen a good solution to allow a data scientist to work locally, but utilize GPUs remotely, without basically just developing remotely (through a VM or Jupyter), or submitting remote jobs (through SLURM or a library specific Kubernetes integration). Scuda is an interesting step towards a better solution for utilizing remote GPUs easily across a wide range of libraries, not just Pytorch and Tensorflow.
- seattleeng 2y ago
  Why is working locally important?
  - theossuary 2y ago
    Working locally still matters, and this is from someone who normally works in tmux/nvim. When working on vision and 3D ML work, being able to quickly open a visualizer windows is imperative to understanding what's going on. For Gaussian Splatting, point cloud work, SLAM, etc. you have to have access to a desktop environment to see visualizations; they very rarely work well remotely (even if they have some Jupyter support). Working remotely, when having to use a desktop environment is painful, no matter the technology. The best tice come up with is using tmux/vim and sunshine/moonlight, but even still I'd rather just have access to everything locally.
- kkielhofner 2y ago
  > I don't think NCCL + Deepspeed/FSDP are really an alternative to Scuda I put "remote" in quotes because they're not direct equivalents but from a practical standpoint it's the alternate current approach. > they all require the models in question to be designed for distributed training. They also require a lot of support in the libraries being used. IME this has changed quite a bit. Between improved support for torch FSDP, Deepspeed, and especially HF Accelerate wrapping of each with transformer models it's been a while since I've had to put much (if any) work in. That said if you're running random training scripts it likely won't "just work" but given larger models becoming more common I see a lot more torchrun, accelerate, deepspeed, etc in READMEs and code. > This has been a struggle for data scientists for a while now. I haven't seen a good solution to allow a data scientist to work locally, but utilize GPUs remotely, without basically just developing remotely (through a VM or Jupyter) Remotely, as in over the internet? 400gb ethernet is already too slow vs PCIe5 x16 (forget SXM). A 10gb internet connection is 40x slower (plus latency impacts). Remote development via the internet with scuda would be impossibly and completely uselessly slow.

theossuary 2y ago

I don't think NCCL + Deepspeed/FSDP are really an alternative to Scuda, as they all require the models in question to be designed for distributed training. They also require a lot of support in the libraries being used. This has been a struggle for data scientists for a while now. I haven't seen a good solution to allow a data scientist to work locally, but utilize GPUs remotely, without basically just developing remotely (through a VM or Jupyter), or submitting remote jobs (through SLURM or a library specific Kubernetes integration). Scuda is an interesting step towards a better solution for utilizing remote GPUs easily across a wide range of libraries, not just Pytorch and Tensorflow.

seattleeng 2y ago
Why is working locally important?
- theossuary 2y ago
  Working locally still matters, and this is from someone who normally works in tmux/nvim. When working on vision and 3D ML work, being able to quickly open a visualizer windows is imperative to understanding what's going on. For Gaussian Splatting, point cloud work, SLAM, etc. you have to have access to a desktop environment to see visualizations; they very rarely work well remotely (even if they have some Jupyter support). Working remotely, when having to use a desktop environment is painful, no matter the technology. The best tice come up with is using tmux/vim and sunshine/moonlight, but even still I'd rather just have access to everything locally.
kkielhofner 2y ago
> I don't think NCCL + Deepspeed/FSDP are really an alternative to Scuda I put "remote" in quotes because they're not direct equivalents but from a practical standpoint it's the alternate current approach. > they all require the models in question to be designed for distributed training. They also require a lot of support in the libraries being used. IME this has changed quite a bit. Between improved support for torch FSDP, Deepspeed, and especially HF Accelerate wrapping of each with transformer models it's been a while since I've had to put much (if any) work in. That said if you're running random training scripts it likely won't "just work" but given larger models becoming more common I see a lot more torchrun, accelerate, deepspeed, etc in READMEs and code. > This has been a struggle for data scientists for a while now. I haven't seen a good solution to allow a data scientist to work locally, but utilize GPUs remotely, without basically just developing remotely (through a VM or Jupyter) Remotely, as in over the internet? 400gb ethernet is already too slow vs PCIe5 x16 (forget SXM). A 10gb internet connection is 40x slower (plus latency impacts). Remote development via the internet with scuda would be impossibly and completely uselessly slow.