While interesting, I think this model is only viable for a certain subset of computational workloads. AF2 took like two weeks of computation on 200 TPUs to tra

While interesting, I think this model is only viable for a certain subset of computational workloads. AF2 took like two weeks of computation on 200 TPUs to train. I’m not sure if they used a data parallel or model parallel distributed training approach, but I don’t imagine either would scale particularly well to a globally distributed network of worker machines in a folding@home analogue...