3 ms·
The library they're using is literally called Hivemind [0]. I'm interested to see how the approach they're using differs from what we use in federated learning
by techwizrd 4y ago
The library they're using is literally called Hivemind [0]. I'm interested to see how the approach they're using differs from what we use in federated learning or gossip learning.
> Hivemind is a PyTorch library for decentralized deep learning across the Internet.
0: https://github.com/learning-at-home/hivemind https://github.com/learning-at-home/hivemind
- borzunov 4y agoA Hivemind/Petals dev here. As far as I understand, most federated learning methods can't efficiently train very large models (with billions of parameters) because they repeat some calculations on many peers and/or involve excess communication. In contrast, the training methods implemented in Hivemind struggle to minimize compute and communication but don't provide data privacy guarantees. This is mostly okay for LLMs, since they are trained on public data scraped from the Internet anyway.