3 ms·
Actually, nope! SGD is embarrassingly parallel when it's HOGWILD! https://arxiv.org/abs/1106.5730 https://arxiv.org/abs/1106.5730. Formally, the hogwild and ser
by inciampati 5y ago
Actually, nope! SGD is embarrassingly parallel when it's HOGWILD! https://arxiv.org/abs/1106.5730 https://arxiv.org/abs/1106.5730. Formally, the hogwild and serial implementations are indistinguishable. This has a lot of citations but it's such an easy trick that no door path beating seems to have happened. The "trick" is just to make model updates atomic, and then each SGD step can happen in parallel without coordination or locks.
- modeless 5y agoSorry, all Hogwild does is remove locks. It still needs constant global communication (global shared memory is assumed in the paper). That's not embarrassingly parallel.
- kettleballroll 5y agoAs far as I know, hogwild doesn't really scale to very large systems and doesn't get used in practice, but maybe that's just because it's most useful on CPUs, which are inefficient for other reasons