3 ms·
Most of the information about this is in this PDF (I hate when people publish interesting information exclusively in PDFs): https://raw.githubusercontent.com/No
by simonw 2y ago
Most of the information about this is in this PDF (I hate when people publish interesting information exclusively in PDFs): https://raw.githubusercontent.com/NousResearch/DisTrO/main/A_Preliminary_Report_on_DisTrO.pdf https://raw.githubusercontent.com/NousResearch/DisTrO/main/A...
I converted it to Markdown (using Gemini 1.5 Pro) and pasted it into a Gist here: https://gist.github.com/simonw/46a33d66e069efe5c10b63625fdabb4e#a-preliminary-report-on-distro https://gist.github.com/simonw/46a33d66e069efe5c10b63625fdab...
From the abstract:
> Training large scale neural networks typically involves sharing gradients between all accelerators, which necessitates specialized, high-speed interconnects. To address this, we introduce DisTrO, a family of architecture-agnostic and network-agnostic distributed optimizers that reduces the inter-GPU communication requirements by four to five orders of magnitude without relying on amortized analysis, enabling low-latency training of large neural networks on slow internet bandwidths with heterogeneous networking hardware.
This could be a HUGE deal.
Currently if you want to train giant LLMs you need a big pile of GPUs in the same location as each other due to the amount of information that needs to shuffle between them during training.
If DisTrO works as intended, it will be possible to train models using GPUs in different places - potentially enabling SETI@home style training where thousands of people with gaming PCs at home could donate their GPU time to a large training effort.
Their tweet about this has more: https://twitter.com/NousResearch/status/1828121648383566270 https://twitter.com/NousResearch/status/1828121648383566270
> Nous Research is proud to release a preliminary report on DisTrO (Distributed Training Over-the-Internet) a family of architecture-agnostic and network-agnostic distributed optimizers that reduces the inter-GPU communication requirements by 1000x to 10,000x without relying on amortized analysis, and matches AdamW+All-Reduce in convergence rates. This enables low-latency training of large neural networks on slow internet bandwidths with heterogeneous networking hardware.
> DisTrO can increase the resilience and robustness of training LLMs by minimizing dependency on a single entity for computation. DisTrO is one step towards a more secure and equitable environment for all participants involved in building LLMs.
> Without relying on a single company to manage and control the training process, researchers and institutions can have more freedom to collaborate and experiment with new techniques, algorithms, and models. This increased competition fosters innovation, drives progress, and ultimately benefits society as a whole.
- liuliu 2y agoAs much as I liked the team, there is really no information other than the loss graph :(
- az226 2y agoThat’s not quite true. They also tested benchmarks and compared with an AdamW trained model.
- bugglebeetle 2y agoNot reproducible, so this doesn’t exist.
- arilotter 2y agowe're excited to drop the paper & code and let you run the code for yourself to see otherwise !! coming in the next couple months :)
- nickpsecurity 2y agoLook at the PDF in simonw’s first link. There’s plenty of information. One part looked like the communication requirements were reduced down to under 100MB. That suggests a communication rate that could be handled by dirt-cheap instances spread across the globe. Like on vast.ai or something.
- liuliu 2y agoGaLore can do that too (if you transplant it). There are similar methods prior LLM era doing the same. They are not quite there on the loss graph side though (I am actually unsure about GaLore, but both FLoRA and ReLoRA were not quite there on loss graph side).
- atlas_hugged 2y agoAwwww snaaaaap, decentralized skyneeet here we goooooo! Kidding of course, but very exciting breakthrough if this pans out.