3 ms·
Imagine if it was possible to combine this with homomorphic encryption into something like AirDrop for LLMs! ("Sorry, I don't know how to answer that – but you
by lxgr 4y ago
Imagine if it was possible to combine this with homomorphic encryption into something like AirDrop for LLMs!
("Sorry, I don't know how to answer that – but you can try getting closer to a bunch of other people running the app on their device and ask again!")
- chatmasta 4y agoSounds like a literal hive mind!
- techwizrd 4y agoThe library they're using is literally called Hivemind [0]. I'm interested to see how the approach they're using differs from what we use in federated learning or gossip learning. > Hivemind is a PyTorch library for decentralized deep learning across the Internet. 0: https://github.com/learning-at-home/hivemind https://github.com/learning-at-home/hivemind
- borzunov 4y agoA Hivemind/Petals dev here. As far as I understand, most federated learning methods can't efficiently train very large models (with billions of parameters) because they repeat some calculations on many peers and/or involve excess communication. In contrast, the training methods implemented in Hivemind struggle to minimize compute and communication but don't provide data privacy guarantees. This is mostly okay for LLMs, since they are trained on public data scraped from the Internet anyway.
- sebzim4500 4y agoHomomorphic encryption has such an enormous overhead that it would never be faster than just running the model locally. Or probably on your wristwatch for that matter.