5 ms·

The intention behind it is pretty good. Best of luck to them. I wonder if I can donate computing power to this remotely. Like the old SETI or protein folding t

The intention behind it is pretty good. Best of luck to them. I wonder if I can donate computing power to this remotely. Like the old SETI or protein folding things. Use idle CPU to calculate for the network. Otherwise the estimates I have seen on how much it would take to train these models are enormous.

teruakohatu 6y ago
TensorFlow supports distributed training with a client server model.
- newswasboring 6y ago
  Does it also solve the problem of everyone having different hardware?
  - londons_explore 6y ago
    It does. For most models, your home broadband would be far too slow though.
    - newswasboring 6y ago
      Is it because they will have to communicate back errors during training? I forgot that training these models is more of a global task than proteins folding. In that sense this is less parallelizable over the internet.
      - londons_explore 6y ago
        Yes, and also activations if your GPU is too small to fit the whole model. The minimum useful bandwidth for that stuff is a few gigabits...
    - emteycz 6y ago
      What about some kind of sharding, parts of the computation that could be executed in isolation for a longer period of time?
      - Filligree 6y ago
        An ongoing research problem. OpenAI would certainly like being able to use smaller GPUs, instead of having to fit the entire model into one.
        jne2356 6y ago
        GPT-3 does not fit in any one GPU that exists at present. It's already spread out across multiple GPUs.
danwills 6y ago
Yeah! sounds great, if I could easily run a SETI-at-home style thing to contribute to training a (future) model similar to GPT-x, but with the result freely available to play with, I reckon i'd do it. Could even be made into a game I'd say! I am totally aware that gpt3 itself can't be run on a regular workstation, but maybe hosting for instances of models from the best/most-interesting training runs could be worked out by crowd-funding?
- colobas 6y ago
  I was just gonna propose something like this. Democratizing large ML models.
chillee 6y ago
The primary issue is that large scale GPU training is primarily dominated by communication costs. Since to some approximation things need to be synchronized after every gradient update, it becomes very quickly quite infeasible to increase the communication cost.
mryab 6y ago
Not directly related, but the Learning@home [1] project aims to achieve precisely that goal of public, volunteer-trained neural networks. The idea is that you can host separate "experts," or parts of your model (akin to Google's recent Switch Transformers paper) on separate computers. This way, you never have to synchronize the weights of the entire model across the participants — you only need to send the gradients/activations to a set of peers. Slow connections are mitigated with asynchronous SGD and unreliable/disconnected experts can be discarded, which makes it more suitable for Internet-like networks. Disclaimer: I work on this project. We're currently implementing a prototype, but it's not yet GPT-3 sized. Some issues like LR scheduling (crucial for Transformer convergence) and shared parameter averaging (for gating etc.) are tricky to implement for decentralized training over the Internet. [1] https://learning-at-home.github.io/ https://learning-at-home.github.io/
- echelon 6y ago
  Do you have a personal Twitter account I can follow? Your career is one I'd like to follow.
  - mryab 6y ago
    Sure! It's @m_ryabinin
    - echelon 6y ago
      Thanks! :D
- gabriel_shatana 6y ago
  Your project looks so interesting. Have u thught of putting the experts on a distributed market where their expertise and work can be exchanged for some token (obvlsy using a blockchain). This would encourage people to host experts in your network and would create value.
  - mryab 6y ago
    Thank you! This is definitely something we should look into in the future (hopefully with community help); as of now, training infrastructure and model convergence are the highest priorities. That said, we welcome all ideas of ways to motivate more volunteers to join the experiments, because Learning@home team comes from a distributed DL background with limited volunteer computing expertise. Also, I believe that for some projects (e.g. GPT-3 replication effort) people would want to join the network regardless of the incentive mechanism, as demonstrated by Leela Chess Zero [1]. [1] http://lczero.org/ http://lczero.org/
jne2356 6y ago
They get this suggestion a lot. There's a section in their FAQ that explains why it's infeasible. https://github.com/EleutherAI/info https://github.com/EleutherAI/info
rijoja 6y ago
You're a genius! How do we get this started. Wouldn't there be so many bitcoin rigs already that could be used for this already? I'd be very willing to spend time and money on this! Could someone set up a chat room or something?

teruakohatu 6y ago

TensorFlow supports distributed training with a client server model.

newswasboring 6y ago
Does it also solve the problem of everyone having different hardware?
- londons_explore 6y ago
  It does. For most models, your home broadband would be far too slow though.
  - newswasboring 6y ago
    Is it because they will have to communicate back errors during training? I forgot that training these models is more of a global task than proteins folding. In that sense this is less parallelizable over the internet.
    - londons_explore 6y ago
      Yes, and also activations if your GPU is too small to fit the whole model. The minimum useful bandwidth for that stuff is a few gigabits...
  - emteycz 6y ago
    What about some kind of sharding, parts of the computation that could be executed in isolation for a longer period of time?
    - Filligree 6y ago
      An ongoing research problem. OpenAI would certainly like being able to use smaller GPUs, instead of having to fit the entire model into one.
      - jne2356 6y ago
        GPT-3 does not fit in any one GPU that exists at present. It's already spread out across multiple GPUs.

danwills 6y ago

Yeah! sounds great, if I could easily run a SETI-at-home style thing to contribute to training a (future) model similar to GPT-x, but with the result freely available to play with, I reckon i'd do it. Could even be made into a game I'd say! I am totally aware that gpt3 itself can't be run on a regular workstation, but maybe hosting for instances of models from the best/most-interesting training runs could be worked out by crowd-funding?

colobas 6y ago
I was just gonna propose something like this. Democratizing large ML models.

chillee 6y ago

The primary issue is that large scale GPU training is primarily dominated by communication costs. Since to some approximation things need to be synchronized after every gradient update, it becomes very quickly quite infeasible to increase the communication cost.

mryab 6y ago

Not directly related, but the Learning@home [1] project aims to achieve precisely that goal of public, volunteer-trained neural networks. The idea is that you can host separate "experts," or parts of your model (akin to Google's recent Switch Transformers paper) on separate computers. This way, you never have to synchronize the weights of the entire model across the participants — you only need to send the gradients/activations to a set of peers. Slow connections are mitigated with asynchronous SGD and unreliable/disconnected experts can be discarded, which makes it more suitable for Internet-like networks. Disclaimer: I work on this project. We're currently implementing a prototype, but it's not yet GPT-3 sized. Some issues like LR scheduling (crucial for Transformer convergence) and shared parameter averaging (for gating etc.) are tricky to implement for decentralized training over the Internet. [1] https://learning-at-home.github.io/ https://learning-at-home.github.io/

echelon 6y ago
Do you have a personal Twitter account I can follow? Your career is one I'd like to follow.
- mryab 6y ago
  Sure! It's @m_ryabinin
  - echelon 6y ago
    Thanks! :D
gabriel_shatana 6y ago
Your project looks so interesting. Have u thught of putting the experts on a distributed market where their expertise and work can be exchanged for some token (obvlsy using a blockchain). This would encourage people to host experts in your network and would create value.
- mryab 6y ago
  Thank you! This is definitely something we should look into in the future (hopefully with community help); as of now, training infrastructure and model convergence are the highest priorities. That said, we welcome all ideas of ways to motivate more volunteers to join the experiments, because Learning@home team comes from a distributed DL background with limited volunteer computing expertise. Also, I believe that for some projects (e.g. GPT-3 replication effort) people would want to join the network regardless of the incentive mechanism, as demonstrated by Leela Chess Zero [1]. [1] http://lczero.org/ http://lczero.org/

jne2356 6y ago

They get this suggestion a lot. There's a section in their FAQ that explains why it's infeasible. https://github.com/EleutherAI/info https://github.com/EleutherAI/info

rijoja 6y ago

You're a genius! How do we get this started. Wouldn't there be so many bitcoin rigs already that could be used for this already? I'd be very willing to spend time and money on this! Could someone set up a chat room or something?