3 ms·
He also points out that AF2 has a large team of 18 coauthors, which is something very rarely seen in academia, where teams are almost always small. In physics,
by ISL 6y ago
He also points out that AF2 has a large team of 18 coauthors, which is something very rarely seen in academia, where teams are almost always small.
In physics, we routinely have author lists of hundreds to thousands of authors on the big experiments. When resources are pooled to make a measurement, author lists can be huge.
University groups can absolutely compete with teams from industry, especially in consortia. The big difference I see between industry and academia in this context at the moment is that academics have to submit a grant proposal and compete for 6-12 months with scientists nationwide for $1M in compute-credits, whereas someone inside a company looking to flex its ML muscles may be able to requisition the same with an email.
- aftbit 6y agoIf computation is the bottleneck, one might imagine that crowdsourced computing like Folding@Home and BOINC could offer an opportunity for less resourced teams to compete.
- rsfern 6y agoWhile interesting, I think this model is only viable for a certain subset of computational workloads. AF2 took like two weeks of computation on 200 TPUs to train. I’m not sure if they used a data parallel or model parallel distributed training approach, but I don’t imagine either would scale particularly well to a globally distributed network of worker machines in a folding@home analogue...
- rcxdude 6y agoSuch resources are only really good for a particular kind of distributed computing: mostly the kind which is easily verifiable, and easily broken down into small embarrassingly parallel chunks which need a lot of computation done relative to their size. Training large machine models is almost the opposite of this: the datasets are huge and don't split up easily, and iterations (which touch the entire model and a chunk of the dataset) are relatively fast. Even with dedicated hardware a lot of the challenge in getting these systems to work effectively is in moving the data around between different compute nodes. Something like folding@home just will not function for such a workload.