3 ms·
Will this enable federated learning? Petals.ml and distributed inference didn't actually work - the technical aspects of what you're doing to enable distributed
by dcreater 2y ago
Will this enable federated learning? Petals.ml and distributed inference didn't actually work - the technical aspects of what you're doing to enable distributed training is well over my head unfortunately
- arilotter 2y agoyou could absolutely use DisTrO for federated learning. The DeMo optimizer on its own doesn't solve the adverserial aspects of training on local-only data, nor does it solve tensor parallelism across devices, so you're still limited to only what fits on your local GPUs, but it does enable distributed data parallelism over the internet at a bandwidth orders of magnitude lower than before.