4 ms·
tl;dr they found a clever way to spread the training across 256 GPUs by synchronising the stochastic gradient descent
by Seanny123 9y ago
tl;dr they found a clever way to spread the training across 256 GPUs by synchronising the stochastic gradient descent
- randyrand 9y agothis seems like the trivial, most obvious way to parallelize training across GPUs. Not, imo, clever. The important bit here is that they've shown that large mini batch sizes still can maintain accuracy if you slow the learning rate.
- Seanny123 9y agovery personally, clever means "well, that would have taken me a long time to figure out and code properly"
- willvarfar 9y agoLots of clever things are simply self-evident in hindsight.
- randyrand 9y agoSomeone asked 5 days ago on HN how to parallelize gpu learning. I had never thought about the problem before, but still came up with and gave this as the most obvious way. Took me maybe 20 seconds to think of. https://news.ycombinator.com/item?id=14510146 https://news.ycombinator.com/item?id=14510146 So the fact that someone with little experience can come up with this 'clever' technique, means either I'm really clever or it's not that clever. I'll go with the latter.
- opportune 9y agoJust because it's simple to explain at a high level doesn't make it trivial. Plenty of theoretically trivial solutions to problems are absolute pains to implement. I mean there are entire companies that at their core solve relatively "trivial" problems but employ huge numbers of engineers. Just because the core concept is simple to explain doesn't mean it's easy.
- randyrand 9y agoSomeone asked the other day how to parallelize gpu learning. I had never thought about the problem before, but still came up with and gave this as the most obvious way. https://news.ycombinator.com/item?id=14510146 https://news.ycombinator.com/item?id=14510146
- eggie5 9y agoit is a very interesting idea, considering how GD updates are inherently a serial operation. I've always wondered how they pulled it off in spark... I've also read about hogwild which allows parallel SGD on sparse datasets...