4 ms·
Long story short, training requires intensive device-to-device communication. Distributed training is possible in theory but so inefficient that it's not worth
by zone411 4y ago
Long story short, training requires intensive device-to-device communication. Distributed training is possible in theory but so inefficient that it's not worth it. Here is a new paper that looks to be the most promising approach yet: https://arxiv.org/abs/2301.11913 https://arxiv.org/abs/2301.11913
- nylonstrung 4y agoWould have to be federated learning to work I think
- sillysaurusx 4y agoIt doesn’t, actually. The model weights can be periodically averaged with each other. No need for synchronous gradient broadcasts. Why people aren’t doing this has always been a mystery to me. Relevant: https://battle.shawwn.com/swarm-training-v01a.pdf https://battle.shawwn.com/swarm-training-v01a.pdf
- telotortium 4y agoYou linked a paper with no results and no conclusion. Perhaps you meant to link a different paper?
- sillysaurusx 4y agoI never finished it.
- hackernewds 4y agoso it is unproven? what is the value of it?
- sillysaurusx 4y agoIt’s how we trained roughly 40 GPT 1.5B models. The technique works; it’s up to you to try it out.
- zone411 4y agoThe abstract mentions fine-tuning, not full pre-training?
- sillysaurusx 4y agoYeah, sorry for not being precise. We used the technique to fine tune around 40 GPT 1.5B models, including the chess one. It was very apparent that the technique was working well. The kiss curve suddenly started dropping dramatically the first day we got it working.
- naasking 4y agoWouldn't other signal propagation approaches, like Forward-Forward, make this easier?
- 6510 4y agoI think the landscape has plenty to think about with few explorers able to wrap their wetware around all of it?