4 ms·
> If there was an effective, distributed means for training LLMs Is it reasonable to believe that this is possible? Distributed training requires extremely hig
by mrshadowgoose 3y ago
> If there was an effective, distributed means for training LLMs
Is it reasonable to believe that this is possible? Distributed training requires extremely high-bandwidth and low-latency interconnect. The internet ain't that.
Believe me, I dearly want the open-source community to "win". A future where only governments have a monopoly on AI research is absolutely terrifying. Given the known parameters though, that future seems inevitable.
- max51 3y ago>Is it reasonable to believe that this is possible? Distributed training requires extremely high-bandwidth and low-latency interconnect. The internet ain't that. The same can be said about piracy and bittorrent. The community torrents is so large that I would bet a lot money they have more computing+bandwidth resources combined than openAI by a wide margin, probably orders of magnitude.
- TeMPOraL 3y agoWhy low latency? Is the latency here a meaningfully annoying limit in training, that can't be reasonably offset by just adding more compute nodes to the network? Latency is the limit in the end, but I feel that there's plenty of easy-ish wins to have in redesigning the architecture and training approach to make it an irrelevant in practice.