4 ms·

DisTrO – a family of low latency distributed optimizers

88 points by SchwKatze 2y ago

deleted 2y ago
[deleted]
simonw 2y ago
Most of the information about this is in this PDF (I hate when people publish interesting information exclusively in PDFs): https://raw.githubusercontent.com/NousResearch/DisTrO/main/A_Preliminary_Report_on_DisTrO.pdf https://raw.githubusercontent.com/NousResearch/DisTrO/main/A... I converted it to Markdown (using Gemini 1.5 Pro) and pasted it into a Gist here: https://gist.github.com/simonw/46a33d66e069efe5c10b63625fdabb4e#a-preliminary-report-on-distro https://gist.github.com/simonw/46a33d66e069efe5c10b63625fdab... From the abstract: > Training large scale neural networks typically involves sharing gradients between all accelerators, which necessitates specialized, high-speed interconnects. To address this, we introduce DisTrO, a family of architecture-agnostic and network-agnostic distributed optimizers that reduces the inter-GPU communication requirements by four to five orders of magnitude without relying on amortized analysis, enabling low-latency training of large neural networks on slow internet bandwidths with heterogeneous networking hardware. This could be a HUGE deal. Currently if you want to train giant LLMs you need a big pile of GPUs in the same location as each other due to the amount of information that needs to shuffle between them during training. If DisTrO works as intended, it will be possible to train models using GPUs in different places - potentially enabling SETI@home style training where thousands of people with gaming PCs at home could donate their GPU time to a large training effort. Their tweet about this has more: https://twitter.com/NousResearch/status/1828121648383566270 https://twitter.com/NousResearch/status/1828121648383566270 > Nous Research is proud to release a preliminary report on DisTrO (Distributed Training Over-the-Internet) a family of architecture-agnostic and network-agnostic distributed optimizers that reduces the inter-GPU communication requirements by 1000x to 10,000x without relying on amortized analysis, and matches AdamW+All-Reduce in convergence rates. This enables low-latency training of large neural networks on slow internet bandwidths with heterogeneous networking hardware. > DisTrO can increase the resilience and robustness of training LLMs by minimizing dependency on a single entity for computation. DisTrO is one step towards a more secure and equitable environment for all participants involved in building LLMs. > Without relying on a single company to manage and control the training process, researchers and institutions can have more freedom to collaborate and experiment with new techniques, algorithms, and models. This increased competition fosters innovation, drives progress, and ultimately benefits society as a whole.
- liuliu 2y ago
  As much as I liked the team, there is really no information other than the loss graph :(
  - az226 2y ago
    That’s not quite true. They also tested benchmarks and compared with an AdamW trained model.
    - bugglebeetle 2y ago
      Not reproducible, so this doesn’t exist.
      - arilotter 2y ago
        we're excited to drop the paper & code and let you run the code for yourself to see otherwise !! coming in the next couple months :)
  - nickpsecurity 2y ago
    Look at the PDF in simonw’s first link. There’s plenty of information. One part looked like the communication requirements were reduced down to under 100MB. That suggests a communication rate that could be handled by dirt-cheap instances spread across the globe. Like on vast.ai or something.
    - liuliu 2y ago
      GaLore can do that too (if you transplant it). There are similar methods prior LLM era doing the same. They are not quite there on the loss graph side though (I am actually unsure about GaLore, but both FLoRA and ReLoRA were not quite there on loss graph side).
- atlas_hugged 2y ago
  Awwww snaaaaap, decentralized skyneeet here we goooooo! Kidding of course, but very exciting breakthrough if this pans out.
logicchains 2y ago
I'd love to believe it's true but I suspect they're overstating the result, or it's a fluke. Presumably teams at large firms like Meta would have put a lot of effort into checking whether not-synchronise-every-step training could match synchronise-every-step training before investing hundreds of millions of dollars into the low-latency, high-throughput network hardware necessary for the latter.
- regularfry 2y ago
  Not if it cost them a month to do so.
- arilotter 2y ago
  We're pretty confident it's not a fluke, and paper + code are the next step, within a couple months. It's not "synchronize every step", but it's "do something every step". We double and triple and quadruple checked our results, to make sure that we are in fact getting results like this while only doing our thing every step, and it really keeps holding up. Don't trust our word for it, though, you'll see when the paper comes out :)
  - RicoElectrico 2y ago
    Um, so why announce something before even a paper with replicable details is available? To put it bluntly, what are we supposed to do with the information? I could be less harsh if this was some grant requirement to release a report before a certain date, but I don't see any grant funding declaration.
    - CuriouslyC 2y ago
      I'm happy to have the project on my radar, and though they could be a bit clearer about the provisional nature of the research I don't think it's wrong to want to hype the potential of it a bit.
    - arilotter 2y ago
      We're excited about the potential and want to find other folks also excited about it that are interested in working for/with us to build things on the foundations of DisTrO! Plus also it's so cool and mind boggling to us that we wanted to share the hype a little bit, it was hard not being able to tell anyone we were working on it
iamronaldo 2y ago
This seems huge no? Couldn't this enable "community based" ai training at home?
- FuckButtons 2y ago
  AFAIK, the main bottleneck on training is memory bandwidth. Distributed gpu compute has multiple orders of magnitude less than an equivalent number of GPUs colocated, because they don’t share a physical bus, but have a network connection instead. This work improves on that, but the fundamental limitations remain.
arjvik 2y ago
There's no information about what this is, beyond a teaser of a loss graph. Really hoping this is something that gets released to the world, not hidden behind closed doors.
- arilotter 2y ago
  Paper & code in the next couple months. We're workin on em :)

88 points by SchwKatze 2y ago

deleted 2y ago

[deleted]

Most of the information about this is in this PDF (I hate when people publish interesting information exclusively in PDFs): https://raw.githubusercontent.com/NousResearch/DisTrO/main/A_Preliminary_Report_on_DisTrO.pdf https://raw.githubusercontent.com/NousResearch/DisTrO/main/A... I converted it to Markdown (using Gemini 1.5 Pro) and pasted it into a Gist here: https://gist.github.com/simonw/46a33d66e069efe5c10b63625fdabb4e#a-preliminary-report-on-distro https://gist.github.com/simonw/46a33d66e069efe5c10b63625fdab... From the abstract: > Training large scale neural networks typically involves sharing gradients between all accelerators, which necessitates specialized, high-speed interconnects. To address this, we introduce DisTrO, a family of architecture-agnostic and network-agnostic distributed optimizers that reduces the inter-GPU communication requirements by four to five orders of magnitude without relying on amortized analysis, enabling low-latency training of large neural networks on slow internet bandwidths with heterogeneous networking hardware. This could be a HUGE deal. Currently if you want to train giant LLMs you need a big pile of GPUs in the same location as each other due to the amount of information that needs to shuffle between them during training. If DisTrO works as intended, it will be possible to train models using GPUs in different places - potentially enabling SETI@home style training where thousands of people with gaming PCs at home could donate their GPU time to a large training effort. Their tweet about this has more: https://twitter.com/NousResearch/status/1828121648383566270 https://twitter.com/NousResearch/status/1828121648383566270 > Nous Research is proud to release a preliminary report on DisTrO (Distributed Training Over-the-Internet) a family of architecture-agnostic and network-agnostic distributed optimizers that reduces the inter-GPU communication requirements by 1000x to 10,000x without relying on amortized analysis, and matches AdamW+All-Reduce in convergence rates. This enables low-latency training of large neural networks on slow internet bandwidths with heterogeneous networking hardware. > DisTrO can increase the resilience and robustness of training LLMs by minimizing dependency on a single entity for computation. DisTrO is one step towards a more secure and equitable environment for all participants involved in building LLMs. > Without relying on a single company to manage and control the training process, researchers and institutions can have more freedom to collaborate and experiment with new techniques, algorithms, and models. This increased competition fosters innovation, drives progress, and ultimately benefits society as a whole.

liuliu 2y ago
As much as I liked the team, there is really no information other than the loss graph :(
- az226 2y ago
  That’s not quite true. They also tested benchmarks and compared with an AdamW trained model.
  - bugglebeetle 2y ago
    Not reproducible, so this doesn’t exist.
    - arilotter 2y ago
      we're excited to drop the paper & code and let you run the code for yourself to see otherwise !! coming in the next couple months :)
- nickpsecurity 2y ago
  Look at the PDF in simonw’s first link. There’s plenty of information. One part looked like the communication requirements were reduced down to under 100MB. That suggests a communication rate that could be handled by dirt-cheap instances spread across the globe. Like on vast.ai or something.
  - liuliu 2y ago
    GaLore can do that too (if you transplant it). There are similar methods prior LLM era doing the same. They are not quite there on the loss graph side though (I am actually unsure about GaLore, but both FLoRA and ReLoRA were not quite there on loss graph side).
atlas_hugged 2y ago
Awwww snaaaaap, decentralized skyneeet here we goooooo! Kidding of course, but very exciting breakthrough if this pans out.

logicchains 2y ago

I'd love to believe it's true but I suspect they're overstating the result, or it's a fluke. Presumably teams at large firms like Meta would have put a lot of effort into checking whether not-synchronise-every-step training could match synchronise-every-step training before investing hundreds of millions of dollars into the low-latency, high-throughput network hardware necessary for the latter.

regularfry 2y ago
Not if it cost them a month to do so.
arilotter 2y ago
We're pretty confident it's not a fluke, and paper + code are the next step, within a couple months. It's not "synchronize every step", but it's "do something every step". We double and triple and quadruple checked our results, to make sure that we are in fact getting results like this while only doing our thing every step, and it really keeps holding up. Don't trust our word for it, though, you'll see when the paper comes out :)
- RicoElectrico 2y ago
  Um, so why announce something before even a paper with replicable details is available? To put it bluntly, what are we supposed to do with the information? I could be less harsh if this was some grant requirement to release a report before a certain date, but I don't see any grant funding declaration.
  - CuriouslyC 2y ago
    I'm happy to have the project on my radar, and though they could be a bit clearer about the provisional nature of the research I don't think it's wrong to want to hype the potential of it a bit.
  - arilotter 2y ago
    We're excited about the potential and want to find other folks also excited about it that are interested in working for/with us to build things on the foundations of DisTrO! Plus also it's so cool and mind boggling to us that we wanted to share the hype a little bit, it was hard not being able to tell anyone we were working on it

iamronaldo 2y ago

This seems huge no? Couldn't this enable "community based" ai training at home?

FuckButtons 2y ago
AFAIK, the main bottleneck on training is memory bandwidth. Distributed gpu compute has multiple orders of magnitude less than an equivalent number of GPUs colocated, because they don’t share a physical bus, but have a network connection instead. This work improves on that, but the fundamental limitations remain.

arjvik 2y ago

There's no information about what this is, beyond a teaser of a loss graph. Really hoping this is something that gets released to the world, not hidden behind closed doors.

arilotter 2y ago
Paper & code in the next couple months. We're workin on em :)