3 ms·

Incredible how much benefit alphafold has brought. And all of that from a less than 100million parameters model. I might be dumb but could they scale it up and

Incredible how much benefit alphafold has brought. And all of that from a less than 100million parameters model. I might be dumb but could they scale it up and make an alphafold 3 with maybe like 10bln params? Would it be a lot better assuming the same training effort is put into it? If it does, can't biotech companies just go nuts and make a 100bln params internal model and have all the protein structures they want?

tiedieconderoga 4y ago
Has there been much research into the idea of distributing these large models across many heterogeneous machines? I'm wondering if there could be a path towards a mix of alphafold and folding@home, with donated idle compute resources being used to train/run the models. Designing for that sort of fragmentation could also make it easier to slowly run oversized models on local machines with swapped memory.
- greggsy 4y ago
  I think volunteer folding will always need to compete with crypto mining, and the appeal may be further diminished if commercial interests are likely to reap the rewards. Perhaps if there was a way to combine mining and folding, to allow participants to somehow gain a share of the output? Eg each folded protein would have a unique hash, which could then be traded? And yes, I hate everything about what I just typed.
- ragebol 4y ago
  There's https://stablehorde.net/ https://stablehorde.net/ running inference only afaik.
- f_devd 4y ago
  Using any current architecture it is infeasible to do backprop (training) due to the massive communication requirements. Inference is possible to do in sharded way but still not as practical as just loading the model weight that are needed on-demand from disk; still a distributed job queue being processed may be beneficial depending on the throughput/costs required by researchers.
- zavrel 4y ago
  There was great work last year on distributed training and inference: Petals: Collaborative Inference and Fine-tuning of Large Models Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, Colin Raffel https://arxiv.org/abs/2209.01188 https://arxiv.org/abs/2209.01188 Would be wonderful to use this host this model LLaMA: Open and Efficient Foundation Language Models https://arxiv.org/abs/2302.13971 https://arxiv.org/abs/2302.13971

tiedieconderoga 4y ago

Has there been much research into the idea of distributing these large models across many heterogeneous machines? I'm wondering if there could be a path towards a mix of alphafold and folding@home, with donated idle compute resources being used to train/run the models. Designing for that sort of fragmentation could also make it easier to slowly run oversized models on local machines with swapped memory.

greggsy 4y ago
I think volunteer folding will always need to compete with crypto mining, and the appeal may be further diminished if commercial interests are likely to reap the rewards. Perhaps if there was a way to combine mining and folding, to allow participants to somehow gain a share of the output? Eg each folded protein would have a unique hash, which could then be traded? And yes, I hate everything about what I just typed.
ragebol 4y ago
There's https://stablehorde.net/ https://stablehorde.net/ running inference only afaik.
f_devd 4y ago
Using any current architecture it is infeasible to do backprop (training) due to the massive communication requirements. Inference is possible to do in sharded way but still not as practical as just loading the model weight that are needed on-demand from disk; still a distributed job queue being processed may be beneficial depending on the throughput/costs required by researchers.
zavrel 4y ago
There was great work last year on distributed training and inference: Petals: Collaborative Inference and Fine-tuning of Large Models Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, Colin Raffel https://arxiv.org/abs/2209.01188 https://arxiv.org/abs/2209.01188 Would be wonderful to use this host this model LLaMA: Open and Efficient Foundation Language Models https://arxiv.org/abs/2302.13971 https://arxiv.org/abs/2302.13971