3 ms·
SETI@home style distributed computation is not suitable for training something like GPT-3, unlike for SETI, the unit of work a node can do before needing to sha
by Voloskaya 6y ago
SETI@home style distributed computation is not suitable for training something like GPT-3, unlike for SETI, the unit of work a node can do before needing to share it's output with the next node is really small, so very fast interconnect between the nodes is needed (Infiniband and NVLink is used in clusters to train it). It would probably take a decade to train such a model over regular internet.
- mitjam 6y agoMaybe a case for a community colocation cloud where I a consumer can buy a system and colocate it in a large data center with great internal networking. Edit: typo
- leogao 6y agoHandling heterogenous (and potentially untrustworthy) systems also adds overhead, not to mention that buying hardware in bulk is cheaper, so it makes the most sense just to raise the money and buy the hardware.
- mirekrusin 6y agoThe problem is potentially solvable as generating solutions takes a lot of GPU time and verifying it is very fast. Aquiring input data may be a problem, but should be possible with dedicated models for this type of computation.
- mirekrusin 6y agoAre there any models/research optimised on working on this kind of small, distributed batches that would fit ie. ~10GB of commodity GPU?