3 ms·
Related: New Training Technique for Highly Efficient AI Methods (2 points, 5 hours ago) https://news.ycombinator.com/item?id=42690664 https://news.ycombinator.
by gnabgib 2y ago
Related:
New Training Technique for Highly Efficient AI Methods (2 points, 5 hours ago) https://news.ycombinator.com/item?id=42690664 https://news.ycombinator.com/item?id=42690664
DiLoCo: Distributed Low-Communication Training of Language Models (46 points, 1 year ago, 14 comments) https://news.ycombinator.com/item?id=38549337 https://news.ycombinator.com/item?id=38549337
- metadat 2y agoThanks for the links, some interesting discussion there. The second article you linked indicates there will still be intense bandwidth requirements during training, shipping around gradient differentials. What has changed in the past year? Is this technique looking better, worse, or the same?
- gaogao 2y agoYeah, high bandwidth requirements still remaining. Over the past year, more research has looked from fully async to restrained cases that allow for geographically distributed compute. Async Local-SGD goes for a more standard training objective comparable with a lockstep training, https://arxiv.org/abs/2401.09135 https://arxiv.org/abs/2401.09135. imo technique is looking better.