4 ms·

That's an interesting question. It actually provides a nice way to parallelized training: Pretrain e.g. the first 3 branch levels, which effectively fragments t

That's an interesting question. It actually provides a nice way to parallelized training: Pretrain e.g. the first 3 branch levels, which effectively fragments the model into 8 separate parts, which you can continue training across 8 independent servers/nodes with no further communication between the nodes. A central server would run the 1st 3 levels and mark parts of the training set that each node has to train on. Maybe you could do this for the whole network and distribute the training in SETI@HOME style all over the world. Hold on, you don't even need to freeze the branches completely: each node could train 1 branch on the path to its leaf node and communicate a change in the branch node to a central server, so you can distribute training without having to pre-freeze the branches. Still would need some pre-training though, and the splits would change slowly, and the attention mechanism could complicate things. Currently distributed neural network training SETI@HOME style looks like a complete pipe dream that nobody is taking seriously. But a smart branching mechanism like this could suddenly make it possible. Folding@home reached 1.5 exaflops, which made it the world's largest supercomputer. Imagine the models we could train that way, they would far surpass whatever OpenAI or Google could train and would be public.

alchemist1e9 3y ago
This! If this becomes true then it’s a game changer. I hope you are correct.
richardw 3y ago
Also steps up the economic benefit of, and therefore demand for, botnets. We really need a solution to bad actors controlling vast amounts of compute.
- klyrs 3y ago
  That ship has sailed, and her name is bitcoin.
  - richardw 3y ago
    If bitcoin keeps the botnets away from world beating AI, worth it.
- optimalsolver 3y ago
  https://en.wikipedia.org/wiki/Hashcash https://en.wikipedia.org/wiki/Hashcash
Geee 3y ago
I am barely understanding, so a stupid question: Does this also mean that it would be possible to train on parallel GPU-poor setup instead of needing lots of GPU memory / bandwidth on one computer?
- dartos 3y ago
  Probably not. The paper talks about this as an inference time optimization
deleted 3y ago
[deleted]
bloopernova 3y ago
Apologies for layman question: how much tera/peta/exa-flops do current models use to train? Well, I'm assuming they'd use whatever they're given, so maybe the question should be "how much less time would training take on a 1.5 exaflops computer?"
- foobiekr 3y ago
  As many as they can afford. A lot of clusters are totally homogeneous, at least within some very large domains, so for a given interconnect and a generation of GPU you know the maximum message latency, the peak sustained pflop rate, and so on but what often matters is some combination of the depreciation-cost-per-time and the watt hours per unit time, where you can sort of approximate both if you ignore the unfortunate realities, which then act as a multiplier. For example, a problem is network issues - and not just scale - as the training sequence often involve billions of cycles of short compute-sync sequences which are bursty (e.g., all-to-all, barrier, compute, barrier, all to all, ...) but between which there isn't enough time to engage low power modes so you're burning $ due to slack and waste. This is true in different ways for a lot of training approaches. You can approximate this, but it's so sensitive to data set size, specific training schedule, etc. that you won't be able to get the most important answer.
dvmazur 3y ago
You should check out Hivemind[1]. It is very similar to what you described except it used MoE for "fragmentation". They have a couple of examples of pre-training in their repo. Hivemind was also used to build Petals[2] but it only supports fine-tuning and inference[3] afaik. [1] https://github.com/learning-at-home/hivemind https://github.com/learning-at-home/hivemind [2] https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals [3] https://chat.petals.dev/ https://chat.petals.dev/

alchemist1e9 3y ago

This! If this becomes true then it’s a game changer. I hope you are correct.

richardw 3y ago

Also steps up the economic benefit of, and therefore demand for, botnets. We really need a solution to bad actors controlling vast amounts of compute.

klyrs 3y ago
That ship has sailed, and her name is bitcoin.
- richardw 3y ago
  If bitcoin keeps the botnets away from world beating AI, worth it.
optimalsolver 3y ago
https://en.wikipedia.org/wiki/Hashcash https://en.wikipedia.org/wiki/Hashcash

Geee 3y ago

I am barely understanding, so a stupid question: Does this also mean that it would be possible to train on parallel GPU-poor setup instead of needing lots of GPU memory / bandwidth on one computer?

dartos 3y ago
Probably not. The paper talks about this as an inference time optimization

deleted 3y ago

[deleted]

bloopernova 3y ago

Apologies for layman question: how much tera/peta/exa-flops do current models use to train? Well, I'm assuming they'd use whatever they're given, so maybe the question should be "how much less time would training take on a 1.5 exaflops computer?"

foobiekr 3y ago
As many as they can afford. A lot of clusters are totally homogeneous, at least within some very large domains, so for a given interconnect and a generation of GPU you know the maximum message latency, the peak sustained pflop rate, and so on but what often matters is some combination of the depreciation-cost-per-time and the watt hours per unit time, where you can sort of approximate both if you ignore the unfortunate realities, which then act as a multiplier. For example, a problem is network issues - and not just scale - as the training sequence often involve billions of cycles of short compute-sync sequences which are bursty (e.g., all-to-all, barrier, compute, barrier, all to all, ...) but between which there isn't enough time to engage low power modes so you're burning $ due to slack and waste. This is true in different ways for a lot of training approaches. You can approximate this, but it's so sensitive to data set size, specific training schedule, etc. that you won't be able to get the most important answer.

dvmazur 3y ago

You should check out Hivemind[1]. It is very similar to what you described except it used MoE for "fragmentation". They have a couple of examples of pre-training in their repo. Hivemind was also used to build Petals[2] but it only supports fine-tuning and inference[3] afaik. [1] https://github.com/learning-at-home/hivemind https://github.com/learning-at-home/hivemind [2] https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals [3] https://chat.petals.dev/ https://chat.petals.dev/