3 ms·

Has there been much research into the idea of distributing these large models across many heterogeneous machines? I'm wondering if there could be a path toward

Has there been much research into the idea of distributing these large models across many heterogeneous machines? I'm wondering if there could be a path towards a mix of alphafold and folding@home, with donated idle compute resources being used to train/run the models. Designing for that sort of fragmentation could also make it easier to slowly run oversized models on local machines with swapped memory.

greggsy 4y ago
I think volunteer folding will always need to compete with crypto mining, and the appeal may be further diminished if commercial interests are likely to reap the rewards. Perhaps if there was a way to combine mining and folding, to allow participants to somehow gain a share of the output? Eg each folded protein would have a unique hash, which could then be traded? And yes, I hate everything about what I just typed.
ragebol 4y ago
There's https://stablehorde.net/ https://stablehorde.net/ running inference only afaik.
f_devd 4y ago
Using any current architecture it is infeasible to do backprop (training) due to the massive communication requirements. Inference is possible to do in sharded way but still not as practical as just loading the model weight that are needed on-demand from disk; still a distributed job queue being processed may be beneficial depending on the throughput/costs required by researchers.
zavrel 4y ago
There was great work last year on distributed training and inference: Petals: Collaborative Inference and Fine-tuning of Large Models Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, Colin Raffel https://arxiv.org/abs/2209.01188 https://arxiv.org/abs/2209.01188 Would be wonderful to use this host this model LLaMA: Open and Efficient Foundation Language Models https://arxiv.org/abs/2302.13971 https://arxiv.org/abs/2302.13971

greggsy 4y ago

I think volunteer folding will always need to compete with crypto mining, and the appeal may be further diminished if commercial interests are likely to reap the rewards. Perhaps if there was a way to combine mining and folding, to allow participants to somehow gain a share of the output? Eg each folded protein would have a unique hash, which could then be traded? And yes, I hate everything about what I just typed.

ragebol 4y ago

There's https://stablehorde.net/ https://stablehorde.net/ running inference only afaik.

f_devd 4y ago

Using any current architecture it is infeasible to do backprop (training) due to the massive communication requirements. Inference is possible to do in sharded way but still not as practical as just loading the model weight that are needed on-demand from disk; still a distributed job queue being processed may be beneficial depending on the throughput/costs required by researchers.

zavrel 4y ago

There was great work last year on distributed training and inference: Petals: Collaborative Inference and Fine-tuning of Large Models Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, Colin Raffel https://arxiv.org/abs/2209.01188 https://arxiv.org/abs/2209.01188 Would be wonderful to use this host this model LLaMA: Open and Efficient Foundation Language Models https://arxiv.org/abs/2302.13971 https://arxiv.org/abs/2302.13971