3 ms·
Is there any framework/system that distributes the work across multiple GPUs on different computers over a network (LAN or WAN)? I'm not concerned much about la
by Nouser76 3y ago
Is there any framework/system that distributes the work across multiple GPUs on different computers over a network (LAN or WAN)? I'm not concerned much about latency or generation time, but would love to train or load up huge models and send jobs to run overnight.
- jph00 3y agoYes, you can run FSDP/QLoRA over multi-node. There's a slurm script in the repo showing how to do it.