3 ms·
If computation is the bottleneck, one might imagine that crowdsourced computing like Folding@Home and BOINC could offer an opportunity for less resourced teams
by aftbit 6y ago
If computation is the bottleneck, one might imagine that crowdsourced computing like Folding@Home and BOINC could offer an opportunity for less resourced teams to compete.
- rsfern 6y agoWhile interesting, I think this model is only viable for a certain subset of computational workloads. AF2 took like two weeks of computation on 200 TPUs to train. I’m not sure if they used a data parallel or model parallel distributed training approach, but I don’t imagine either would scale particularly well to a globally distributed network of worker machines in a folding@home analogue...
- rcxdude 6y agoSuch resources are only really good for a particular kind of distributed computing: mostly the kind which is easily verifiable, and easily broken down into small embarrassingly parallel chunks which need a lot of computation done relative to their size. Training large machine models is almost the opposite of this: the datasets are huge and don't split up easily, and iterations (which touch the entire model and a chunk of the dataset) are relatively fast. Even with dedicated hardware a lot of the challenge in getting these systems to work effectively is in moving the data around between different compute nodes. Something like folding@home just will not function for such a workload.