4 ms·
Author here - in this case we are talking mostly about ~halving the time of a single minibatch in training a variety of standard neural networks. That type of t
by spiantino 11y ago
Author here - in this case we are talking mostly about ~halving the time of a single minibatch in training a variety of standard neural networks. That type of training tends to be very hard to distribute. The servers also have 8 gpus in a single host, which is an additional performance boost and isn't included in the 2x above.
- hiddencost 11y agoI'm curious why this claim (distributed minibatch is hard) continues to be repeated. Am I misreading the state-of-the-art? There was a paper at Interspeech this year [0] where Amazon showed excellent scaling (54X speed up when distributed across 80 nodes) without loss of accuracy. [0] http://www.nikkostrom.com/publications/interspeech2015/strom_interspeech2015.pdf http://www.nikkostrom.com/publications/interspeech2015/strom...