8 ms·
The Hogwild approach mentioned is extremely simple. When you do incremental updates, don't hold locks. You'll rarely conflict, because the number of data points
by pradn 12y ago
The Hogwild approach mentioned is extremely simple. When you do incremental updates, don't hold locks. You'll rarely conflict, because the number of data points to update is usually much higher than the number of threads doing the updates. And even if you do conflict, the effects don't affect accuracy in any noticeable way. (I found this empirically.)
It looks like Microsoft is applying this technique not to stochastic gradient descent, but to building neural networks, in this case.
- liuliu 12y agoActually, I don't quite understand their method. At the end of the day, it is not a gigantic shared memory, as in Hogwild! paper conveniently describes. You need to synchronize slave's model with master model, in model parallelism case, you need to send responses to another machine for further processes. This sounds extremely complicated actually to implement in modern HPC environment (assuming MPI as the main implementation technique). Sorry if this sounds a bit defensive than it is meant to be. I just think that there are much more to the implementation than the simple Hogwild! approach (probably some clever pipelining to cover the communication / computation cost as the Hogwild! approach relaxed consistency of the model). It seems that this year's ImageNet classification competition result will be more interesting than what I expected.