4 ms·
Does this require distributed training? My understanding and experience is that it is not always trivial to get linear training speedups with additional machin
by etaioinshrdlu 8y ago
Does this require distributed training?
My understanding and experience is that it is not always trivial to get linear training speedups with additional machines.
It can be sometimes hard to get any speedup at all.
The difference with gradient descent can be described simply as:
* Single-machine training: take more steps
* Distributed training: take fewer but more confident and accurate steps. This could allow you to take bigger steps (learning rate) as well, but there is a limit to this as well.
They are not equivalent processes and it is an area of active research how to get an equivalent result with distributed training.
- antognini 8y agoI worked on a research project that set out to answer that question: https://arxiv.org/abs/1811.03600 https://arxiv.org/abs/1811.03600 We looked at a bunch of different model architectures and datasets and found that you can get speedups from larger batch sizes up until a certain point, but that point is different for different datasets and architectures. The range of good hyperparameters is also narrower for larger batch sizes, which makes tuning harder.
- gdahl 8y agoTo clarify, we found the range shrink if one trains for a fixed number of epochs and expand if one trains for a fixed number of steps. So tuning can be easier or harder depending on the budget.