3 ms·
this seems like the trivial, most obvious way to parallelize training across GPUs. Not, imo, clever. The important bit here is that they've shown that large mi
by randyrand 9y ago
this seems like the trivial, most obvious way to parallelize training across GPUs. Not, imo, clever.
The important bit here is that they've shown that large mini batch sizes still can maintain accuracy if you slow the learning rate.
- Seanny123 9y agovery personally, clever means "well, that would have taken me a long time to figure out and code properly"
- willvarfar 9y agoLots of clever things are simply self-evident in hindsight.
- randyrand 9y agoSomeone asked 5 days ago on HN how to parallelize gpu learning. I had never thought about the problem before, but still came up with and gave this as the most obvious way. Took me maybe 20 seconds to think of. https://news.ycombinator.com/item?id=14510146 https://news.ycombinator.com/item?id=14510146 So the fact that someone with little experience can come up with this 'clever' technique, means either I'm really clever or it's not that clever. I'll go with the latter.
- opportune 9y agoJust because it's simple to explain at a high level doesn't make it trivial. Plenty of theoretically trivial solutions to problems are absolute pains to implement. I mean there are entire companies that at their core solve relatively "trivial" problems but employ huge numbers of engineers. Just because the core concept is simple to explain doesn't mean it's easy.
- randyrand 9y agoSomeone asked the other day how to parallelize gpu learning. I had never thought about the problem before, but still came up with and gave this as the most obvious way. https://news.ycombinator.com/item?id=14510146 https://news.ycombinator.com/item?id=14510146