4 ms·

Is it possible to distribute AI training like this so that people could donate their idle processing power to creating a truly open source unencumbered LLM?

rtkwe 4y ago
The issue with all these types of projects now is that computers are way better at idling now so there's no longer as much effectively free compute to be harvested. SETI@home made a lot more sense when CPUs consumed roughly the same power regardless of the load.
gradys 4y ago
It’s harder because gradient descent is usually done with near synchronous updates of the parameters across the whole model on all training hosts.
- why_only_15 4y ago
  Yeah, the synchronous updates are a big deal. To get an idea of how much bandwidth this typically takes, Oracle has 1600gbps per node of interconnect with latency very low, guessing in the tens of microseconds. A really good home connection might have 1gbps of interconnect with latency in the tens of milliseconds. The big question is whether we really need all this interconnect -- GPT-JT[1] is a very promising step in this direction. The idea is that we just drop most of the gradient updates and it still works well. Unclear whether this will take off generally -- if it does it would be a huge deal, because 1600gbps of interconnect is very expensive. [1]: https://www.together.xyz/blog/releasing-v1-of-gpt-jt-powered-by-open-source-ai https://www.together.xyz/blog/releasing-v1-of-gpt-jt-powered...
ikmckenz 4y ago
The Hivemind project is just that https://github.com/learning-at-home/hivemind https://github.com/learning-at-home/hivemind
alwayslikethis 4y ago
It's not clear that machine learning models consisting of weights are copyrightable in the first place. They are not the results of human authorship, but are instead essentially produced by a machine trying to optimize a bunch of values.
- mr_toad 4y ago
  If you can get copyright on a photo just by pointing a camera and pressing a button you can probably get copyright on an AI.
  - ROTMetro 4y ago
    I'm guessing you missed this discussion: https://news.ycombinator.com/item?id=35191206 https://news.ycombinator.com/item?id=35191206 No copyright on an AI if it was machine generated, which the training is. Great catch kmeisthax!
kmeisthax 4y ago
Yes. You'd have to package up slices of the training set and the prior epoch as work units and host them. Our hypothetical GPT@home clients download the training set, train just those examples, and then send back their gradients. The server then averages all the gradients together. As far as I can tell this isn't any different conceptually from the distributed inference setup that PyTorch/Huggingface Accelerate/etc support. Of course those were built for cooperating machines owned by the same organization with lots of high-speed I/O. In an @home/BOINC setup, you need to package things up into work units that get sent out to other people's computers, and you don't get anything back until the end. So it's higher-latency and you have to worry about people cheating the system with bogus work. I'm not sure if this would be more efficient or cheaper than just renting out a bunch of AWS spot instances with GPUs in them.
- cypress66 4y ago
  Batches need to be reasonably sized. Llms are usually trained for a single epoch. If you just average the gradients once, it will be awful. The main problem I believe is that you would need extremely high bandwidth. For a 100B model you're looking at 400GB of gradients you need to send each batch. There's a reason why super fast networking is used.
  - deleted 4y ago
    [deleted]
- pxmpxm 4y ago
  Err gradients aren't independently additive like that right?

rtkwe 4y ago

The issue with all these types of projects now is that computers are way better at idling now so there's no longer as much effectively free compute to be harvested. SETI@home made a lot more sense when CPUs consumed roughly the same power regardless of the load.

gradys 4y ago

It’s harder because gradient descent is usually done with near synchronous updates of the parameters across the whole model on all training hosts.

why_only_15 4y ago
Yeah, the synchronous updates are a big deal. To get an idea of how much bandwidth this typically takes, Oracle has 1600gbps per node of interconnect with latency very low, guessing in the tens of microseconds. A really good home connection might have 1gbps of interconnect with latency in the tens of milliseconds. The big question is whether we really need all this interconnect -- GPT-JT[1] is a very promising step in this direction. The idea is that we just drop most of the gradient updates and it still works well. Unclear whether this will take off generally -- if it does it would be a huge deal, because 1600gbps of interconnect is very expensive. [1]: https://www.together.xyz/blog/releasing-v1-of-gpt-jt-powered-by-open-source-ai https://www.together.xyz/blog/releasing-v1-of-gpt-jt-powered...

ikmckenz 4y ago

The Hivemind project is just that https://github.com/learning-at-home/hivemind https://github.com/learning-at-home/hivemind

alwayslikethis 4y ago

It's not clear that machine learning models consisting of weights are copyrightable in the first place. They are not the results of human authorship, but are instead essentially produced by a machine trying to optimize a bunch of values.

mr_toad 4y ago
If you can get copyright on a photo just by pointing a camera and pressing a button you can probably get copyright on an AI.
- ROTMetro 4y ago
  I'm guessing you missed this discussion: https://news.ycombinator.com/item?id=35191206 https://news.ycombinator.com/item?id=35191206 No copyright on an AI if it was machine generated, which the training is. Great catch kmeisthax!

kmeisthax 4y ago

Yes. You'd have to package up slices of the training set and the prior epoch as work units and host them. Our hypothetical GPT@home clients download the training set, train just those examples, and then send back their gradients. The server then averages all the gradients together. As far as I can tell this isn't any different conceptually from the distributed inference setup that PyTorch/Huggingface Accelerate/etc support. Of course those were built for cooperating machines owned by the same organization with lots of high-speed I/O. In an @home/BOINC setup, you need to package things up into work units that get sent out to other people's computers, and you don't get anything back until the end. So it's higher-latency and you have to worry about people cheating the system with bogus work. I'm not sure if this would be more efficient or cheaper than just renting out a bunch of AWS spot instances with GPUs in them.

cypress66 4y ago
Batches need to be reasonably sized. Llms are usually trained for a single epoch. If you just average the gradients once, it will be awful. The main problem I believe is that you would need extremely high bandwidth. For a 100B model you're looking at 400GB of gradients you need to send each batch. There's a reason why super fast networking is used.
- deleted 4y ago
  [deleted]
pxmpxm 4y ago
Err gradients aren't independently additive like that right?